Opsera leverages its DevOps platform and integrations and builds AI agents and frameworks to revolutionize the software delivery management process with a unique approach to automating data orchestration.
Opsera is now part of Databricks’ Built on Partner Program and Technology Partner Program.

The partnership enables:
● AI/ML Model Deployments with Security and Compliance Guardrails: Opsera
ensures that model training and deployment using Databricks infrastructure meets
security and quality guardrails and thresholds before deployment. Proper model training
allows customers to optimize Databricks Mosaic AI usage and reduce deployment risks.

● Schema Deployments with Rollback Capabilities: Opsera facilitates controlled
schema deployments in Databricks with built-in rollback features for enhanced flexibility
and confidence. Customers gain better change management and compliance tracking
and reduce unfettered production deployments, leading to increased adoption of
Databricks and enhanced value of automation pipelines.

“The development of advanced LLM models and Enterprise AI solutions continues to fuel an
insatiable demand for data,” said Torsten Volk, Principal Analyst at Enterprise Strategy Group.
“Partnerships between data management and data orchestration vendors to simplify the
ingestion and ongoing management of these vast flows of data are necessary responses to
these complex and extremely valuable AI efforts.”

Additional benefits of the Opsera and Databricks partnership include:
● Powerful ETL (Extract, Transform, Load) Capabilities: Databricks’ Spark-based
engine enables efficient ETL from various sources into a centralized data lake. This
empowers Opsera to collect and orchestrate vast amounts of data, increasing developer
efficiency and accelerating data processing efficiency.
● Scalable and Flexible Data Intelligence Platform: Databricks’ Delta UniForm and
Unity Catalog provide a scalable, governed, interoperable, and reliable Data Lakehouse
solution, enabling Opsera to orchestrate large volumes of structured and unstructured
data efficiently.
● Advanced Analytics and ML: Databricks Mosaic AI’s integrated machine learning
capabilities allow Opsera to efficiently build and deploy AI/ML models for predictive
analytics, anomaly detection and other advanced use cases.
● Seamless Integration: Databricks integrates seamlessly with Opsera’s existing
technology stack, facilitating smooth data flow and enabling end-to-end visibility of the
DevOps platform.

The post Opsera and Databricks partner to automate data orchestration appeared first on SD Times.

A major drawback to achieving DDD success often occurs when organizations treat data governance as a broad, enterprise-wide initiative rather than an iterative, use-case-focused process. In this way, the approach often leads to governance shortcomings such as a lack of context, where generic policies overlook the specific requirements of individual domains and fail to address unique use cases effectively. Adopting governance across an entire organization is usually time-consuming and complex, which leads to delays in realizing the benefits of DDD. Additionally, employees tend to resist large-scale governance changes that seem irrelevant to their daily tasks, impeding adoption and effectiveness. Inflexibility is another concern, as enterprise-wide governance programs are difficult to adapt to evolving business needs, which can stifle innovation and agility.

Another common challenge when applying domain-driven design involves the concept of bounded context, which is a central pattern in DDD. According to Fowler, bounded content is the focus of DDD’s strategic design, which is all about dealing with large models and teams. This approach deals with large models by dividing them into different Bounded Contexts and being explicit about their interrelationships, thereby defining the limits within which a model applies. 

However, real-world implementations of bounded contexts present challenges. In complex organizations, domains often overlap, making it difficult to establish clear boundaries between them. Legacy systems can exacerbate this issue, as existing data structures may not align with newly defined domains, creating integration difficulties. Many business processes also span multiple domains, further complicating the application of bounded contexts. Traditional organizational silos, which may not align with the ideal domain boundaries, add another layer of complexity, leading to inefficiencies.

Developing well-defined domains is also problematic, as it requires a substantial time commitment from both technical and business stakeholders. This can result in delayed value realization, where the long lead time to build domains delays the business benefits of DDD, potentially undermining support for the initiative. Business requirements may evolve during the domain-building process, necessitating constant adjustments and further extending timelines. This can strain resources, especially for smaller organizations or those with limited data expertise. Furthermore, organizations often struggle to balance the immediate need for data insights with the long-term benefits of well-structured domains.

Making consistent data accessible

Data democratization aims to make data accessible to a broader audience, but it has also given rise to what is known as the “facts” problem. This occurs when different parts of the organization operate with conflicting or inconsistent versions of data. This problem often stems from inconsistent data definitions, and without a unified approach to defining data elements across domains, inconsistencies are inevitable. Despite efforts toward democratization, data silos may persist, leading to fragmented and contradictory information. A lack of data lineage further complicates the issue, making it difficult to reconcile conflicting facts without clearly tracking the origins and transformations of the data. Additionally, maintaining consistent data quality standards becomes increasingly challenging as data access expands across the organization. 

To overcome these challenges and implement domain-driven design successfully, organizations should start by considering the following five steps:

1. Focus on high-value use cases: Prioritize domains that promise the highest business value, enabling quicker wins, which can build momentum for the initiative. 
2. Embrace iterative development: This is essential so organizations should adopt an agile approach, starting with a minimal viable domain, and refining it based on feedback and evolving needs. 
3. Create cross-functional collaboration: Between business and technical teams. This is crucial throughout the process, ensuring that domains reflect both business realities and technical constraints. Investing in metadata management is also vital to maintaining clear data definitions, lineage, and quality standards across domains, which is key to addressing the “facts” problem. 
4. Develop a flexible governance framework: That is adaptable to the specific needs of each domain while maintaining consistency across the enterprise.

To balance short-term gains with a long-term vision, organizations should begin by identifying key business domains based on their potential impact and strategic importance. Starting with a pilot project in a well-defined, high-value domain can help demonstrate the benefits of DDD early on. It also helps businesses to focus on core concepts and relationships within the chosen domain, rather than attempting to model every detail initially.

Implementing basic governance during this phase lays the foundation for future scaling. As the initiative progresses, the domain model also expands to encompass all significant business areas. Cross-domain interactions and data flows should be refined to optimize processes, and advanced governance practices, such as automated policy enforcement and data quality monitoring, can be implemented. Ultimately, establishing a Center of Excellence ensures that domain models and related practices continue to evolve and improve over time.

By focusing on high-value use cases, embracing iterative development, fostering collaboration between business and technical teams, investing in robust metadata management, and developing flexible governance frameworks, organizations can successfully navigate the challenges of domain-driven design. Better yet, the approach provides a solid foundation for data-driven decision-making and long-term innovation.

As data environments grow increasingly complex, domain-driven design continues to serve as a critical framework for enabling organizations to refine and adapt their data strategies, ensuring a competitive edge in a data-centric world.

The post Five steps to successfully implement domain-driven design appeared first on SD Times.

Key advantages include enhanced decision-making agility through real-time analytics, proactive management through predictive insights, and operational efficiency via task automation. This automation allows managers to focus on strategic initiatives, while the data-driven transparency of DSS fosters stakeholder trust and collaboration. Overall, AI-driven DSS is reshaping project management, driving operational excellence, and ensuring success in today’s data-driven landscape.

Fig 1: Convergence of Machine Learning Backend and Analytics Dashboard Frontend: A Synoptic Representation of Data-driven Decision Support in Contemporary Applications.

From Data Insights to Strategic Decision

In modern project management, converting data into useful insights starts by gathering and verifying data from various sources. Machine Learning (ML) tools then analyze this data to find patterns and anomalies. Regression models, like Linear Regression, are great for estimating project costs and durations. Meanwhile, neural networks offer detailed analyses for more complex situations and risk assessments. Together, these data collection and ML analysis methods form a strong foundation, improving accuracy and providing the strategic vision needed for project success. Let’s delve into some of these techniques in context of a project management software.

Linear Regression is useful for predicting continuous outcomes, such as project costs, based on several factors. On the other hand, Logistic Regression predicts binary outcomes, like project success or failure, using past data. While Linear Regression is seen as simpler, understanding its basic principles and results is crucial. Logistic Regression, while similar, requires a deep understanding of its classifications.

In time series forecasting, ARIMA excels in interpreting time-related data, whether it is tracking project progress or resource use. However, mastering ARIMA’s parameters can be challenging. Decision Trees, which combine classification and regression, identify key project factors. Techniques like Random Forest enhance this precision but can be complex and resource-intensive. Neural Networks, such as RNNs and LSTMs, are ideal for analyzing sequences, making them great for tracking project trends. However, they require a thorough understanding, careful adjustments, and abundant data. Clustering Algorithms, like K-Means, group project elements based on similarities, aiding in resource allocation. Hierarchical Clustering reveals deeper data structures and project relationships. While K-Means is simpler, deciding on the number of clusters can be tricky. Hierarchical Clustering, though insightful, can be resource-intensive with large datasets. For detecting anomalies, tools like Isolation Forest are invaluable, highlighting unexpected project deviations. Isolation Forests strike a commendable balance between efficiency and accuracy, tailored for intricate datasets yet relatively simple to deploy. Additionally, using Natural Language Processing (NLP) for tasks like Text Classification and Sentiment Analysis offers insights into project feedback and stakeholder opinions. The complexity of these NLP tasks varies, from basic analyses like those that can be facilitated by packages like NLTK or spaCy to intricate endeavors, especially when addressing specialized domains.

In summary, the integration of these machine learning techniques into project management software equips managers with a wealth of insights derived from historical data, enabling them to make more informed and strategic decisions. By leveraging the power of advanced analytics and predictive modeling, project leaders can anticipate challenges, optimize resource allocation, and foresee potential bottlenecks. However, the efficacy of these techniques is contingent upon several factors, including the unique characteristics of the project, the quality and granularity of available data, as well as the specific analytical requirements. Hence, selecting the most appropriate ML methods tailored to the project’s distinct context and objectives is paramount for deriving actionable insights and achieving desired outcomes.

DSS capabilities of 20 Project Management Tools 

In the modern business environment, Microsoft Project (MSP) rules supreme due to its smooth compatibility with Microsoft tools and advanced analytics via Power BI. Jira by Atlassian is recognized for its agile capabilities and ability to work with multiple plugins, offering adaptability but possibly leading to fragmented setups. Smartsheetmerges the simplicity of spreadsheets with project oversight, presenting visual dashboards; yet, advanced analytics might call for additional BI tool integration. Asana provides straightforward project tracking, but in-depth insights might require supplementary integrations. Trello by Atlassian offers an intuitive Kanban board, with its native analytics often enhanced by Power-Ups or third-party integrations. Basecamp emphasizes communication, offering foundational analytics suitable for modest projects, while larger endeavors might lean towards third-party solutions. Monday.com offers a visually appealing interface with integrated project tracking, although advanced analytics could demand BI integrations. Wrike combines task handling with basic reporting; its adaptability is a highlight, but detailed analytics might need users to explore further integrations. Adobe Workfront addresses enterprise-level demands with comprehensive work management and reporting, potentially being too robust for smaller teams. Clarizen targets repetitive projects with advanced functionalities. It offers advanced project management with customizable analytics. Its focus on custom insights positions it as a potent tool but may pose challenges in terms of complexity. Notion serves as a collaborative platform, merging task management with note-taking. Its versatility is evident, yet comprehensive analytics might require third-party enhancements. ClickUp positions itself as an all-in-one platform with task management, docs, goals, and chat. It can be utilized to streamline project tasks: auto-generate subtasks, condense comments, and autonomously update projects with its AI manager. Its broad feature set is complemented by integrated reporting but might require deeper integrations for advanced analytics. Airtable melds spreadsheet ease with database functionalities; its advanced analytics typically benefit from integrations. Airtable is available on mobile devices and allows for integrations with third-party applications like Slack and Google Drive. Redmine is an open-source tool which offers fundamental project management; for enriched analytics, plugins are needed. Podio provides adaptable work management, facilitating custom workflows. While its analytics are varied, third-party tools can augment its capabilities. Teamwork emphasizes teamwork, merging task and project management, and though its reporting is insightful, deeper analytics might call for more integrations. LiquidPlanner employs predictive planning, showcasing dynamic views with real-time data; however, intricate analytics might still need external tools. Zoho Projects, under the Zoho umbrella, offers holistic project management with in-built reporting; for detailed analytics, other Zoho offerings or integrations might be considered. Targetprocess is for agile management with tailored visualizations; nevertheless, wider analytical demands might drive integration needs. Planview is crafted for enterprise portfolio oversight, providing comprehensive reporting that suits intricate organizational needs.

To sum it up, while these tools offer AI-enhanced features, the scope and sophistication of their Decision Support System capabilities differ. Organizations should evaluate their specific needs, considering built-in features, integrations, or a blend of both to address their Decision Support System requirements effectively. Moreover, this presents a significant chance for these tools to evolve and introduce innovative features and offerings as they transition to the next versions.

Conclusion

In AI research, future decision support systems (DSSs) are expected to employ advanced reinforcement learning models for dynamic decision-making. AI-driven DSSs in project management offer data-driven insights, predictive analytics, and tailored recommendations, elevating decision-making quality. As AI evolves, DSS capabilities will further refine, providing more context-aware solutions for project management challenges. Thus, integrating AI-driven DSSs becomes crucial for achieving operational excellence and sustained project success in today’s complex landscape.

The post Decision Support Systems: Transforming Project Management Software in the New Era appeared first on SD Times.

Renovate helps ensure the security and currency of applications by scanning software to identify external dependencies and automating updates to the latest versions. 

According to Mend.io, while the free Renovate Community Edition and Renovate CLI work well for smaller development environments, they can cause delays for companies with a large number of repositories. The Renovate Enterprise Edition solves this problem by providing unlimited horizontal scaling of server resources, enabling organizations to process numerous repositories simultaneously and ensuring optimal responsiveness for developers.

“Keeping dependencies up to date is one of the most effective ways to reduce technical debt and avoid software vulnerabilities, especially as most companies rely heavily on external dependencies,” said Rhys Arkins, vice president of product at Mend.io. “Mend Renovate Enterprise Edition offers a commercially supported version of Renovate built with the power to help developers handle enterprise-scale needs.”

Benefits of Mend Renovate Enterprise Edition include automatic dependency updates, highly responsive interactivity, reduced technical debt, better code quality, and more. 

Mend Renovate Enterprise Edition and Mend Renovate Community Edition are both self-hosted container-based applications, providing the ability for companies to retain the control that is needed for those with stringent internal security measures. 

Both editions include a job scheduler and webhooks. The job scheduler provides automation for Renovate, while webhooks trigger Renovate tasks in response to important activities like package file updates or PR merges. 

Mend ensures that Renovate Enterprise includes an up-to-date, stable version of the open-source Renovate CLI, ensuring that new Renovate features are continuously integrated while maintaining reliability.

Additional details are available here. 

The post Mend.io launches enterprise version of dependency management tool, Renovate appeared first on SD Times.

Recent laws such as the General Data Protection Regulation (GDPR) instituted in Europe, and the California Consumer Privacy Act in the United States have put limits on collecting and sharing data without the person’s consent. And while enforcement at first was lax, more companies have been hit with fines for not following the regulation.

“The idea of GDPR is good, because the idea is to protect everybody’s personalized data, because you and me, you want to know where your data is used and why it’s used, where it’s coming from and what is recorded about you,” explained Cagdas Gandar, the managing director at the Germany office of data company Melissa. “Before, people had the liberty to just take that data and do whatever they want with your information.”

That, he added, is why there is GDPR. “It became the normal way of life for marketing companies, and for some companies, it meant also that they are not in business anymore – especially companies who collected data without any permission, or consent, and were selling this data,” Gandar said.

He did acknowledge that these regulations have been burdensome on a lot of companies, “and it has hurt them.” But the need for rules about how to collect and work with data overrode the need for marketers.

There seems to be a cat-and-mouse game between marketers and the public, in that marketers want to capture names, while most people just want to get the content without providing any information about themselves. One way to game the system is to input an incorrect name, such as “Mickey Mouse.” Another is to sign up for a disposable email address, which is only valid for 10 minutes or so – giving the person time to get the information they want without any way for the company to follow up with them. 

Part of this is due to the huge volumes of emails people get each day. With these restrictions on data collection and use, one would expect a decrease in the amount of emails people receive from marketers. And Gandar said he is seeing that in Europe.”Maybe you just want to read news or a whitepaper because you’re interested in a topic, but you want to decide by yourself when you want to get in touch with that company to get more information, so that the ball is in your court and you don’t get triggered from everywhere…Everybody calls, but it’s certainly a lot less.”

One way around this is through the use of a data minimization strategy. The idea is to collect just enough personal information to satisfy a request, and to keep the data only as long as it takes to fulfill that request. Beyond that, using personalization marketers can ensure that the people in their database are only receiving emails that align with the person’s expressed areas of interest. 

Unfortunately for many marketers, this will reduce the number of names they can put in their funnels – but the upside is that these are real people with real interests in the subject. Gandar said, “At the end of the day, if you ask for my perfect personal preference, I’d prefer 30 good leads to 270 that are not good. We have to just change our mindset a little bit, that it’s not about quantity anymore. It’s just about quality. It’s also more economical just working on 30, than working on 270 requests, prospects, or leads.”

The key to successful data utilization for marketing? Visibility and transparency, according to Gandar, who related this story. “There was this media house, where they also have different products, different magazines, where you can subscribe. And I thought, it’s really interesting, so I signed up for all of them. And I received a lot of emails, and then at one point, I couldn’t handle it anymore. And this is the problem of all the people, I think you’re getting just too many emails, you don’t have time to read it, and your mailbox gets too big. Then I went through to their website, logged in and clicked the opt-in option. And I could see all my subscriptions, which was really nice. And then I could decide what I want to still receive, and which ones I can unsubscribe from. So more visibility and more transparency was a really nice solution presented there.”

The post Marketers face challenges with data management appeared first on SD Times.

Code Dx has integrated Secure Code Warrior into its automated vulnerability management and correlation platform to offer contextual training specific to an organization’s vulnerabilities. 

“Oftentimes, there are thousands of results in a project and there’s no way for security to triage all of them while keeping up with the speed of DevOps. The Code Dx platform effectively solves that problem,” said Utsav Sanghani, the director of product management at Code Dx. “While fixing problems is critical, preventing coding errors that result in vulnerabilities can only be addressed through consistent, integrated contextual training offered by organizations like Secure Code Warrior.”

Octopai introduces Data Lineage XD

Octopai announced its new multidimensional data lineage platform, Data Lineage XD, to help enterprises achieve an in-depth view of their data flow so that they can get more value from their data assets. 

The platform’s three layers of data management include cross-system lineage, inner-system lineage and end-to-end column lineage.

With the new release, Octopai aims to enhance collaboration for BI teams, democratize data management and also to simplify the onboarding process for new employees. 

New Microsoft Teams features announced

Microsoft announced the general availability of new personal features in Teams to help family and friends communicate in more innovative ways. 

The Together mode feature enables people to turn regular video calls into ones where people share a virtual environment such as a family lounge, coffee shop or a summer resort.

Microsoft also helps people manage tasks directly within their group chats by creating shared to-do lists and assigning tasks to various members of the group. 

Additional details are available here.

Apache weekly update

Last week, the Apache Software Foundation (ASF) saw the release of HttpComponents Client 5.1 GA, which supports encryption with the HTTPS protocol, adds pluggable socket factories and TLS strategies and more. 

Additional releases from the ASF last week included UIMA uimaFIT 3.2.0, Log4cxx 0.12.0, ActiveMQ 5.15.15 and 5.16.2, SkyWalking Kong 0.1.1, Tomcat 8.5.66, 9.0.46 and 10.0.6 as well as MyFaces Core 2.0.25.

Also, Apache Trafodion, the webscale SQL-on-Hadoop solution enabling transactional or operational workloads on Apache Hadoop is now retired due to inactivity. 

Additional details on all of the releases are available here.

The post SD Times news digest: Code Dx and Secure Code Warrior on Project Better Code, Octopai’s Data Lineage XD, and new Microsoft Teams features appeared first on SD Times.

According to the foundation, achieving top-level status means that the project graduated from the Apache Incubator and has demonstrated that it’s community and products have been well-governed under the ASF’s meritocratic process and principles.

The project is dedicated for both streaming and batch data ecosystems and can integrate hundreds of terabytes and thousands of datasets per day by making it easier to ingest, replicate, and organize lifecycle management processes across different types of environments.

The project also simplifies data lake creation by supporting simple transformations and enabling organization within the lake through compaction, partitioning and deduplication.

Users can also benefit from the life cycle and compliance management of data within the lake that includes data retention and fine-grain data deletions, the ASF explained in a blog post.

“Apache Gobblin supports deployment models all the way from a single-process standalone application to thousands of containers running in cloud-native environments, ensuring that your data plane can scale with your company’s growth,” said Shirshanka Das, the founder and CTO at Acryl Data, a member of the Apache Gobblin Project Management Committee.

Gobblin originated at LinkedIn 2014, was open-sourced in 2015, and entered the Apache Incubator in 2017. Apache Gobblin software is released under the Apache License v2.0 and is overseen by a self-selected team of active contributors to the project.

The post Apache Gobblin now top-level project appeared first on SD Times.

Nim 1.4 was the latest release of the language in October, which brought a new major version of the beta-grade package manager Nimble, v0.12.0. It also featured ORC management, Nim’s all-new cycle collector based on ARC, which has all of the advantages or ARC except for determinism, according to the Nim developers in a blog post.

Nim also hit a major milestone by crossing 1500 available Nim packages, and the number of submitted packages this year saw a 35% growth over previous years. 

Postman API Hack announced
Postman’s new hackathon is beginning on Monday, Jan. 25th and offering $100,000 in cash prizes. The winner will be announced at Postman Galaxy which takes places between February 2nd to 4th. 

“Our goal with the Postman API Hack is to highlight the amazing things that become possible by harnessing the power of APIs, and to showcase developers’ work to our community of more than 13 million users and 500,000 organizations. We’re excited to see what participants will come up with, and how they will have an impact on the world,” said Abhinav Asthana, the co-founder and CEO of Postman.

Additional details on the hackathon are available here.

TIBCO acquires Information Builders (ibi) 
The acquisition will add ibi’s data management and analytics capabilities to the advanced TIBCO Connected Intelligence platform.

Ibi has data quality, preparation, and integration products that are now being added to the TIBCO Any Data Hub and TIBCO Responsive Application Mesh. 

“This represents a significant opportunity for TIBCO and ibi as customers strive to become data-first enterprises. There is tremendous potential for any platform that can integrate and manage data to create intelligent workflows for employees, partners, and customers,” said Howard Dresner, the chief research officer at Dresner Advisory Services.

The post SD Times news digest: A recap of Nim in 2020, Postman API Hack announced, and TIBCO acquires Information Builders appeared first on SD Times.

“Digital business needs are evolving faster than ever, now demanding agile responses to changing market conditions, deeper connections between people and technology, and quicker time to market. This calls for responsive, interoperable, and rapidly discoverable digital assets,” said Randy Menon, senior vice president and general manager of connect and TIBCO Cloud.

The new building blocks are part of a partnership with robotic process automation (RPA) provider WorkFusion.

The building block includes:

• TIBCO Cloud integration to monitor apps in a single view, whether they are on cloud, hybrid, or on-premise architectures. The integration also includes a unified dashboard experience and the addition of a marketplace as well as native integration with WorkFusion RPA.
• TIBCO Cloud Mesh for creating and discovering assets such as APIs, flows, events or integrations
• TIBCO BusinessEvents 6 for contextual event processing in a cloud-native world. It includes support for embedded technologies such as Apache Kafka, Apache Cassandra and Apache Ignite
• TIBCO BPM Enterprise 5 that can be deployed natively as containers on cloud platforms.

“Digital business requires fast, smart, connected decisions and processes to execute and win at scale. Rapidly connecting people, processes, and data helps businesses engineer transformative customer experiences that are seamless, highly engaging, responsive, and personalized,” said Sam Fahmy, chief marketing officer for WorkFusion.

Additionally, the company announced the TIBCO Any Data Hub at the conference to provide a data management blueprint for distributed data environments. The hub is designed to reduce the complexity of inconsistent data and unify tools and approaches.

“In a perfect world with perfect systems and never-changing architectures, there would be no need for tools and technologies like master data management and data virtualization. Instead, in the world we live in, organizations look to solutions that will help manage an overwhelming influx of data,” said Mark Palmer, senior vice president for engineering at TIBCO. “We’ve seen a massive expansion in data and an evolution of the systems and architectures that generate and control this data. The TIBCO Any Data Hub blueprint simplifies the entire process for customers by unifying the tools and approaches needed to create a complete and cohesive data picture, making a meaningful business impact.”

As part of the data hub, the company announced TIBCO Data Virtualization 8.3 with an expanded set of data adapters and the ability to connect to over 300 data sources.

Lastly, the company revealed the TIBCO Hyperconverged Analytics solution for real-time analytics for data-driven businesses. The release includes TIBCO Spotfire 11 and TIBCO Cloud Data Streams. Together, the solutions aim to accelerate insights and actions for businesses. With TIBCO Spotfire 11, users will get visual analytics, data science and streaming analytics capabilities through a single environment. The TIBCO Cloud Data Streams is a real-time streaming BI platform that detects trends and patterns as they occur.

“With TIBCO Hyperconverged Analytics, we compress the time from business events to analysis, insight, decision and action for all kinds of data. Proven results are astounding, and we see extreme value generation across many business sectors,” said Michael O’Connell, chief analytics officer for TIBCO.

The post TIBCO expands on distributed data management and responsive application mesh vision appeared first on SD Times.

One of xMatters’ high-profile projects ran into this situation last year. The project, a new feature that allows the end users to get reminders of coming on-call shifts, seemed straightforward. However, as our team began to scratch the surface, they realized the complexity of the project. This included crazy amounts of shift data, as well as the unlimited combinations of data. The project was pushed so many times, we got to a point where we couldn’t push it anymore. 

Test in production, a.k.a. ‘train on race day’
After we exhausted our last resort, we made the important decision to test the project on production. Here’s how the scene unfolded:

Q: “Wait a minute, did you say test in production? Or do you actually mean test in production-like data?”

A: “The best place with production-like data is in production, so we test in production.”

Q: “Doesn’t that sound too risky to run the project in production without proper performance testing? It’s like running a marathon without training.”

A: “The risks are calculated. It’s like training for a marathon on race day, with preparation.”

Before I explain how we made this work, let’s take a look at my experience training for my first marathon. Back in 2016, not long after I accomplished my first 10K race, I decided to give a marathon a shot. As an engineer trained to approach the problem in a very logical manner, I conducted a technical spike, otherwise known as  a Google search. I realized after this search that the most important training I could implement for a full marathon would be to build my endurance. This meant I needed to get to a point where I could complete anywhere from 32K-35K (roughly 20 to 22 miles) in a single run. However, I did have a constraint here—I really didn’t have the time. As a busy parent of two, I couldn’t afford to go on three-hour-long runs; not even occasionally. I came up with a plan:

1. While I could not afford to participate in 30K runs, I would train up to 10K, but with a longer pace.
2. I needed to figure out the areas I needed to work on most. 
3. The best way to know a race is by doing it as often as possible. And I did just that. 

In 2017, one year after my venture into distance running, I ran my first marathon. I managed to walk/limp the last 10K across the finish line, with a 4+-hour finish time. This result was not ideal. However, I was able to get data that I wasn’t able to previously see from any of my training runs, including the following:

1. I did the first half in 1:42, so it built my confidence that I could do a half marathon in 100 minutes.
2. The second half is is much more difficult than the first. I needed to ease into the first half and keep my pace lower.
3. Mostly importantly, preventing cramping needed to become my main focus. I needed to ensure I have proper hydration techniques, fuel and increased strength training.

Similar to my marathon experience, the marathon project we at xMatters were working on had multiple constraints. There was a hard deadline we weren’t able to push anymore, or we would risk the project being cancelled, and the features weren’t finished. Performance testing also hadn’t been done yet, as it required ample time that we just didn’t have. To meet the deadline, we decided to test in production. There were some risks with this including the potential for a bad user experience with a half-baked feature, flooded downstream services and outage, and other services on the same cluster being impacted if CPU or RAM is going out of the capacity.

To minimize the risk, we had to have risk control measures in place including:

1. Extensive monitoring: We set up extensive monitoring to ensure all matrices were in a safe range. At the same time, we gathered valuable data on traffic and usage patterns.
2. Disabled live notification: The live notification portion of the feature was disabled. It was just micro brains doing their calculations in the back end without actually bothering end users.
3. Feature turned on in canary fashion: The feature was turned on in the canary fashion, e.g., 5% of users in one cluster on day one, 10% on day two, etc.

The result was quite eye-opening. We discovered that the usage pattern was quite different from what we expected. Some of the traffic peak was from the source of an epic data sync that we hadn’t planned for. It was clear that some of the expensive API calls caused performance problems. 

Just like my marathon training, testing in production did the trick! Identifying and knowing pain points was integral to achieving good progress. It made it easier for us to figure out which areas we needed to focus on the most. Actions like improving the performance by implementing caching were taken to resolve these problems. We wouldn’t have been able to figure out these areas just from performance testing, no matter how well it might have been defined. It helped us to have a good understanding of usage patterns, while directing us to areas that needed more work. Additionally, it uncovered a lot of unknowns and built the team’s confidence in delivering the feature.

The last push
Back to my marathon running. I didn’t make another attempt to run a marathon for quite a while. Even after almost two years, if someone asked me if I was ready for another 26.2 marathon, I would say, “no.” I’m NEVER ready! When I did decide to register for another marathon, my brain decided for me that a half would be plenty. Two weeks before race-day, I asked myself, “How many more times will you have the opportunity to run a race like this?” I guess the answer is not many. I realized, it’s important to start now; getting TO the start line is half the battle. That final push convinced me to train for another full marathon. Ready or not, I just ran it. And the result was not as bad as I originally anticipated. Compared to the first attempt, my pace improved from 9:40min/Mile to 7:55min/mile.

Similarly, in the final stage of any engineering project delivery, the team may need a final push. The last few miles are always the hardest, especially after all the preparation that has come before to get the team to where it is. Project managers need to make it clear that it’s time to get the project finalized, no matter what it takes. I have witnessed my team struggle on that daunting “wall”  of a project, but I’ve also witnessed their determination to cross the finish line. When the team first started, it didn’t seem feasible to get the project out on time, especially given so many constraints. With determination, willingness to go above and beyond and time management, we accomplished our goals and completed the project. 

One of the greatest marathon runners, Eliud Kipchoge, always said, “No human is limited.” A team, if empowered properly and with proper risk management, can make the impossible possible. 

The post <span class="sdt-premium">premium</span> How training to run a marathon is like bringing a marathon project to the finish line appeared first on SD Times.