Opsera leverages its DevOps platform and integrations and builds AI agents and frameworks to revolutionize the software delivery management process with a unique approach to automating data orchestration.
Opsera is now part of Databricks’ Built on Partner Program and Technology Partner Program.

The partnership enables:
● AI/ML Model Deployments with Security and Compliance Guardrails: Opsera
ensures that model training and deployment using Databricks infrastructure meets
security and quality guardrails and thresholds before deployment. Proper model training
allows customers to optimize Databricks Mosaic AI usage and reduce deployment risks.

● Schema Deployments with Rollback Capabilities: Opsera facilitates controlled
schema deployments in Databricks with built-in rollback features for enhanced flexibility
and confidence. Customers gain better change management and compliance tracking
and reduce unfettered production deployments, leading to increased adoption of
Databricks and enhanced value of automation pipelines.

“The development of advanced LLM models and Enterprise AI solutions continues to fuel an
insatiable demand for data,” said Torsten Volk, Principal Analyst at Enterprise Strategy Group.
“Partnerships between data management and data orchestration vendors to simplify the
ingestion and ongoing management of these vast flows of data are necessary responses to
these complex and extremely valuable AI efforts.”

Additional benefits of the Opsera and Databricks partnership include:
● Powerful ETL (Extract, Transform, Load) Capabilities: Databricks’ Spark-based
engine enables efficient ETL from various sources into a centralized data lake. This
empowers Opsera to collect and orchestrate vast amounts of data, increasing developer
efficiency and accelerating data processing efficiency.
● Scalable and Flexible Data Intelligence Platform: Databricks’ Delta UniForm and
Unity Catalog provide a scalable, governed, interoperable, and reliable Data Lakehouse
solution, enabling Opsera to orchestrate large volumes of structured and unstructured
data efficiently.
● Advanced Analytics and ML: Databricks Mosaic AI’s integrated machine learning
capabilities allow Opsera to efficiently build and deploy AI/ML models for predictive
analytics, anomaly detection and other advanced use cases.
● Seamless Integration: Databricks integrates seamlessly with Opsera’s existing
technology stack, facilitating smooth data flow and enabling end-to-end visibility of the
DevOps platform.

The post Opsera and Databricks partner to automate data orchestration appeared first on SD Times.

Companies can use it to govern structured and unstructured data, and machine learning models, notebooks, dashboards, and files. 

It offers interoperability with any data format and compute engine and supports all of the major cloud platforms. With Unity Catalog, companies can manage data from a number of sources in one place, including sources like MySQL, PostgreSQL, Amazon Redshift, Snowflake, Azure SQL, Azure Synapse, and Google BigQuery, to name a few.

Companies can define access policies once and then apply them across different clouds and platforms, simplifying access management and governance.  

Unity Catalog was first created at Databricks in 2021 as an offering for its customers, and it is being open sourced so that more companies can benefit from it. 

“We’re excited to open source Unity Catalog and release the code,” said Ali Ghodsi, co-founder and CEO of Databricks. “We’ll continue to evolve the open standard in close collaboration with our partners.”

Matt Dugan, VP Data Platforms, AT&T, added: “With the announcement of Unity Catalog’s open sourcing, we are encouraged by Databricks’ step to make lakehouse governance and metadata management possible through open standards. The flexibility to utilize interoperable tools with our data and AI assets, with consistent governance, is core to the AT&T data platform strategy.”

Read about other Open-Source Projects of the Week…

• OpenAI .NET library
• Apache Pinot, an analytics engine for large amounts of data
• Flow-IPC, a tool for making in-memory transfers of data, images, and videos between applications

The post Unity Catalog – SD Times Open Source Project of the Week appeared first on SD Times.

The company hopes that by releasing this model, it will further democratize access to AI and enable its customers to build their own models based on their own data. 

According to Databricks, the new model, DBRX, outperforms the current open source LLMs using the standard benchmarks. It also beats out GPT 3.5 on several benchmarks as well. 

It was created by Mosaic AI, trained on NVIDIA DGX Cloud, and built on the MegaBlocks open source project. 

“At Databricks, our vision has always been to democratize data and AI. We’re doing that by delivering data intelligence to every enterprise — helping them understand and use their private data to build their own AI systems. DBRX is the result of that aim,” said Ali Ghodsi, co-founder and CEO at Databricks.

Dirk Groeneveld, principal software engineer at Allen Institute for Artificial Intelligence (AI2), added: “We’re at an important inflection point for AI that requires a community of researchers, engineers and technologists to better understand it and drive meaningful innovation. This is why our team at AI2 is deeply committed to advancing the science of Generative AI through open model development and are excited to see new models like DBRX bringing greater transparency, accessibility and collaboration to the industry.”

The post Databricks releases new open LLM appeared first on SD Times.

This acquisition aims to enhance Databricks’ capabilities by enabling Databricks to efficiently replicate data from a variety of databases and SaaS applications into its Lakehouse Platform. According to Databricks, this is significant because Data Lakehouse Platforms are essential for enterprise data and AI, but the process of ingesting data from different sources is currently complex and costly. Arcion’s technology will streamline this data ingestion process and enhance Databricks’ offerings.

Data Lakehouse Platforms have become the standard for enterprise data and AI, but their value depends on the data they contain, according to Databricks in a post. 

Ingesting data from existing databases and applications has been a challenge. With Arcion’s technology, Databricks can provide a scalable, cost-effective solution for this data ingestion process. It will leverage Arcion’s scalable CDC engine and connectors for over 20 enterprise databases and data warehouses, making it easier to integrate this data into the Lakehouse Platform while maintaining enterprise-grade security and governance.

The acquisition is seen as a strategic move to address the problem of siloed systems that many businesses face. A recent survey from MIT Technology Review Insights revealed that numerous companies have multiple separate systems, hindering their data and AI efforts.

“To build analytical dashboards, data applications, and AI models, data needs to be replicated from the systems of record like CRM, ERP, and enterprise apps to the Lakehouse,” said Ali Ghodsi, co-founder and CEO of Databricks. “Arcion’s highly reliable and easy-to-use solution will enable our customers to make that data available almost instantly for faster and more informed decision-making. Arcion will be a great asset to Databricks, and we are excited to welcome the team and work with them to further develop solutions to help our customers accelerate their data and AI journeys.”

The post Databricks to acquire change data capture platform Arcion appeared first on SD Times.

Databricks Model Serving offers automatic optimization for LLM Serving, delivering high-performance results without the need for manual configuration. According to Databricks, it’s the first serverless GPU serving product built on a unified data and AI platform, allowing users to create and deploy GenAI applications seamlessly within a single platform, covering everything from data ingestion to model deployment and monitoring.

Databricks Model Serving simplifies the deployment of AI models, making it easy even for users without deep infrastructure knowledge. Users can deploy a wide range of models, including natural language, vision, audio, tabular, or custom models, regardless of how they were trained (from scratch, open-source, or fine-tuned with proprietary data). 

Just log your model with MLflow, and Databricks Model Serving will automatically prepare a production-ready container with GPU libraries like CUDA and deploy it to serverless GPUs. This fully managed service handles everything from managing instances, maintaining version compatibility, to patching versions. It also automatically adjusts instance scaling to match traffic patterns, saving on infrastructure costs while optimizing performance and latency.

Databricks Model Serving has introduced optimizations for serving large language models (LLM) more efficiently, resulting in up to a 3-5x  reduction in latency and cost. To use Optimized LLM Serving, you simply provide the model and its weights, and Databricks takes care of the rest, ensuring your model performs optimally. 

This streamlines the process, allowing you to concentrate on integrating LLM into your application rather than dealing with low-level model optimization. Currently, Databricks Model Serving automatically optimizes MPT and Llama2 models, with plans to support additional models in the future.

The post Databricks introduces public preview of GPU and LLM optimization support for Databricks Model Serving appeared first on SD Times.

One of the announcements was that the company added new features to Lakehouse AI, which is a data-centric platform for building generative AI applications.  

One of the new features is Vector Search, which allows developers to use embedded search when developing generative AI solutions, which helps improve the accuracy of responses. 

They have also added a low-code way to fine-tune LLMs and a curated list of open-source models for getting started with generative AI. 

The company also announced new capabilities in Lakehouse Federation in Unity Catalog, which is a data and AI governance solution for data lakehouses. It now has new querying capabilities that allow customers to consolidate and map data assets from all the platforms they are using.

Customers can also now set up consistent access policies for their data assets and push those policies to other data warehouses. 

“We’re giving organizations access to all of the data they need through one system, which will lead to more innovation — and the best part about that innovation is that it doesn’t sacrifice security,” said Matei Zaharia, co-founder and chief technologist at Databricks. “By enabling customers to easily apply the rules consistently across platforms and track data usage, we’ll help them meet compliance requirements while pushing their businesses forward.”

And finally, the company also announced the preview for LakehouseIQ, which is a natural language interface that learns about an organization’s unique data, culture, and operations using generative AI. This enables AI to provide responses that are specifically tailored to that organization because it understands their specific business jargon and context. 

According to Databricks, LakehouseIQ learns from things like schemas, documents, queries, lineage, notebooks, and BI dashboards. 

Databricks believes this new offering will enable every employee to gain insights from internal data, not just the data scientists. It uses Unity Catalog for governance to ensure that employees have access to the right data and don’t have access to data they shouldn’t. 

“LakehouseIQ will help democratize data access for every company to improve better decision-making and accelerate innovation. With LakehouseIQ, an employee can simply write a question and find the data they need for a project, or get answers to questions relevant to their company’s operations. It removes the roadblocks inherent in traditional data tools and doesn’t require programming skills,” said Ali Ghodsi, co-founder and CEO at Databricks. “Every employee knows the questions to ask to improve their day-to-day work and ultimately their business. With LakehouseIQ, they have the power to quickly and accurately discover the answers.”

The post Databricks announces new AI-powered features in Lakehouse AI, Unity Catalog, and more appeared first on SD Times.

For example, the Cloudflare connection allows sharing from Databricks to Cloudflare R2, which is Cloudflare’s distributed object storage offering, eliminating the need to manage complex data transfers or duplications and ensures that customers are always sharing the most updated data sets. 

“We are in the midst of an AI revolution rooted in data,” said Matthew Prince, co-founder and CEO, Cloudflare. “Cloudflare R2 provides an amazing value proposition for companies that suffer from vendor lock-in, and instead ensures developers retain the power to choose where to move and use their data. The combination of Cloudflare’s massive global network and zero egress storage, along with Databricks’ powerful sharing and processing capabilities, will give our joint customers the fastest, most secure, and most affordable data sharing capabilities across the globe.”

Customers can share data, AI models, and notebooks directly with companies that support Delta Sharing without expensive fees or the need to replicate their data. Delta Sharing provides an easy way to manage permissions as well. 

According to Databricks, the lack of an open standard for sharing data has stunted the creation of a universal secure data exchange, and customers were required to replicate their data across multiple platforms, clouds, and regions if they wanted to sharte it. Delta Sharing allows data sharing with any company that supports the protocol.

“Without an open standard for secure data exchange across organizations, companies find it highly time-consuming to collaborate, requiring export, replication and maintenance of data across many software platforms,” said Matei Zaharia, co-founder and CTO at Databricks. “Delta Sharing provides the first open protocol for sharing data across diverse computing platforms, clouds and regions. Today’s announcements show just how much demand there is for this in the industry, with multiple major technology vendors joining the ecosystem. We are excited about how this will push open interchange forward and help all of our customers collaborate more easily.”

The post Databricks reveals new Delta Sharing partners: Cloudflare, Dell, Oracle, Twilio appeared first on SD Times.

According to the company, this new extension will allow developers to write their code locally utilizing the editing capabilities of VS Code, connect to Databricks clusters and run code remotely, and use the software development best practices of source code control, unit testing, and CI/CD right from their IDE. 

This comes as the first of several planned releases and updates geared towards teams who have standardized on using IDEs for their development processes. 

Databricks stated that it has constructed a new team to focus specifically on the breadth of the developer ecosystem, and it will be rolling out support for additional IDEs as well as other tools that allow developers to have full access to the data lakehouse from third party products. 

“With all your data in one place, you can utilize appropriate-scaled clusters for queries, visualizations, or any other analysis you need. Train machine learning models and deploy jobs to production so that anyone in your organization can see and use data to make decisions, all within VS Code,” the company wrote in a blog post.

Because users can manage  Databricks objects to be  inside VS Code natively, it prevents context switching between applications by enabling them to remain in their IDE.

Visual Studio Code extension preview is now available for download here through the Visual Studio Code Marketplace. To learn more, read the documentation. 

The post Databricks announces Visual Studio Code extension appeared first on SD Times.

Through this integration, Databricks users will be able to quickly provision a new dbt Cloud trial that is already pre-connected to their Databricks account. 

According to dbt Labs, this is helpful for customers who are looking to quickly get a feel for what dbt Labs and Databricks can achieve together through a streamlined and pre-configured workflow. 

“The dbt Community has long had interest in a joint solution with Databricks, and thanks to this accelerating partnership, we’re now able to deliver a turnkey experience to Databricks customers looking to experience dbt Cloud,” said Margaret Francis, chief product officer at dbt Labs. “It’s a natural fit – with dbt as the transformation framework running on top of a unified lakehouse, data teams have access to an open platform with incredibly deep community support.”

This news comes following a series of milestones for both companies, such as Databricks’ development of a dedicated dbt-Databricks adapter and gaining more than 1,000 members of the Databricks and Spark channel with the dbt Community Slack.

“dbt running on Databricks has made modeling accessible directly to business analysts. It all lives in one place and it’s all access controlled, so we don’t have to worry about writing to a separate data warehouse or a separate cloud… Having everyone in the same environment and accessing the same version of the same data, every time, is huge,” said Filippe Felisola Caso, business analytics manager at the Brazilian prop-tech company, Loft.

The post dbt Labs now available on Databricks Partner Connect appeared first on SD Times.

Integrations with Databricks partners Fivetran, Labelbox, Microsoft Power BI, Prophecy, Rivery, and Tableau are initially available to customers, with Airbyte, Blitzz, dbt Labs, and several others coming in the months ahead. 

In addition, with Databricks Partner Connect, users are enabled to discover new, pre-validated solutions from Databricks partners that complement their business needs. With this, users can easily expand their lakehouse into every corner of their data ecosystem in order to solve current or future challenges.

OpenAI’s API now available with no waitlist

OpenAI revealed that its API is now available for developers in supported countries to sign up and start experimenting without a waitlist. Over the past year, OpenAI has made many improvements to it’s API, including the Instruct Series models that adhere better to human instructions, specialized endpoints for more truthful question-answering, and a free content filter to help developers mitigate abuse.

Other changes to the API include an improved Playground, resulting in easier prototyping with models, an example library with many prompts to help developers get started, and Codex, a new model that translates natural language into code.

Logz.io unveils observability updates 

Logz.io, an open source observability platform for modern DevOps teams, announced several updates to its observability platform at its annual ScaleUP user conference. The combination of updates brings users a wide range of advanced and mission-critical capabilities in order to support today’s unified full stack observability.

Available later this year, the updates include unified dashboards, anomaly detection, service performance monitoring, security event management, and OpenSearch. In addition, Logz.io announced that it will expand its operations to Japan before the end of the year.

New Kubernetes certification from The Cloud Native Computing Foundation

The Linux Foundation, the nonprofit enabling innovation through open source, and The Cloud Native Computing Foundation, which builds sustainable ecosystems for cloud native software, have announced that the Kubernetes and Cloud Native Associate (KCNA) exam, originally announced last month, is now generally available for enrollment and scheduling. 

In addition, a new online training course, Kubernetes and Cloud Native Essentials, has been released to both prepare individuals for entry level cloud roles and to sit for the KCNA exam. KCNA is made up of a multiple-choice certification exam designed to test entry-level knowledge and skills in Kubernetes as well as the wider cloud native ecosystem. 

The KCNA exam is broken up into different sections, including Kubernetes fundamentals, container orchestration, cloud native architecture, cloud native observability, and cloud native application delivery. 

The post SD Times news digest: Databricks launched Partner Connect; OpenAI’s API now available with no waitlist; Logz.io unveils observability updates appeared first on SD Times.