Data Science & Analytics Geology & Geoscience Transfer

Q&A with Dr. Mark Zoback: The Approach to Unconventionals

Every well that gets drilled is an opportunity to gain more insight and understanding. The team behind believes in building tools for people to use the data they have to draw the right conclusions. How can machine learning aid geomechanics? Learn more about the approach in the above video.

Data Science & Analytics Geology & Geoscience Transfer

Q&A with Mark Zoback: Unconventional Asset Development

What are the opportunities and challenges with unconventionals? What can geoscience offer to unconventional development? What role can data science play? For a brief discussion, watch this video of Founder and CEO Dr. Troy Ruths with Stanford University Professor of Geophysics, and Technical Adviser, Dr. Mark Zoback.

Data Science & Analytics Geology & Geoscience Transfer

Synthetic Well Log: Reconstructing Bad Hole Well Log Results with Machine Learning

Loss of contact with the borehole.  Cave-ins.  Sub-optimal mud types.  Recording errors.  Unit conflicts.  Equipment failure.  A number of things can cause a Well Log to have bad hole readings.  Perhaps the Caliper log indicates a series of unreliable borehole sections, and an expert has flagged them.  Perhaps the expert has run an outlier detection algorithm to identify aberrant well log readings. 

What next?  What to do?  Re-logging the well is often cost prohibitive.

We’ve built a Synthetic Well Log tool in Spotfire that uses machine learning to help replace those bad hole values with more accurate ones.

Our tool uses the theory behind academic studies (e.g. An Artificially Intelligent Technique to Generate Synthetic Geomechanical Well Logs for the Bakken Formation, Synthetic well logs generation via Recurrent Neural Networks, Generating Synthetic Well Logs by Artificial Neural Networks (ANN) Using MISO-ARMAX Model in Cupiagua Field) and supports several machine learning algorithms (Random Forest, Gradient Boosting, and Support Vector Machines).  The algorithms ingest the curves which do not have data integrity issues in order to predict more accurate values of the missing or faulty curve.

Jumping into the machine learning arena might feel daunting, so we developed this tool to help Geo experts with the process.  Better yet, our tool doesn’t just reach into a black box and hand back a reconstructed well log.  The Synthetic Well Log tool:

  1. Works with the Geo expert to build an imputation model
  2. Lets that person examine modeling validation metrics
  3. Displays the predictions and reconstructed logs next to the original for a sanity check    
  4. Exports the chosen reconstructed model

Our tool puts reconstructed curves right next to the originals so they pass the expert’s eye test as well as the modeling diagnostics.

Data Science & Analytics Geology & Geoscience Transfer and Environmental Stewardship

Oil and Gas companies are aware of their environmental responsibilities, the financial re-directing of governmental decarbonization initiatives, and the growing perception towards climate change. Responding to these concerns, supermajors are investing in battery innovation and renewable energy resources. The acceptance by the O&G community that climate change must be addressed has increased consumer trust as indicated by the steady climb in the 2018 Edelman Trust Barometer since 2014. (

But there’s more to the environmental question than adherence to regulations and alternative fuels. O&G companies realize that resource productivity—getting the most out of a well, isn’t just economically important; it’s a vital part of environmental stewardship.

Renewable resources still need time to mature. Under the Obama administration, Mark Zoback, professor of Geophysics at Stanford University and Technical Advisor to, served on a panel to address the environmental impact of shale gas production. In a Stanford Report article, he noted that “the global energy system is so huge that even if we move as quickly as possible to develop renewable energy sources such as wind, water and solar, we will need to continue using fossil fuels until mid-century.” (

With that stretch of time looming ahead of us, O&G companies are faced with the difficult tasks of (1) figuring out how to effectively fracture the low permeability shale strata to release cleaner burning natural gas and (2) how to recover oil that’s being left in the ground.

As a bridge to our green energy future, natural gas provides a transition alternative that produces fewer pollutants than either coal or oil. Globally, the amount of shale gas is enormous, enough to provide a 100-year buffer at current consumption to the time when those alternative energy sources are ready. Particularly for the creation of electrical energy, natural gas is replacing coal and reducing the carbon footprint by more than 20% since 2006. (See also

Then there’s that tough to extract oil still lying untouched in fields across the world. According to experts, that residual represents two-thirds of the oil in known fields. (

There are many reasons for this large untouched percentage including the rate of extraction in a well, poor geology, well spacing, well construction, or poor hydraulic fracturing methodology.

Both natural gas extraction and left behind oil can be addressed. The data is there for understanding the oil and gas field, but the complexity of the factors to optimize shale extraction requires a multi-faceted, multi-layered approach.

To answer this need, collaborates with Dr. Mark Zoback to embed geomechanics in a platform that bundles data conditioning with machine learning and visualizations to enable analytics at the well, section, and asset level. enables O&G companies to determine the individual elements of a well plan that have the greatest impact on production, boosting the amount of oil extracted from each well and optimizing the correct approach for unconventional drilling.

By increasing the recovery from a well, O&G companies answer the call to environmental stewardship, giving us the buffer we need to move towards a world of alternative energy sources.

Data Science & Analytics Transfer

What Do Cancer Models Have to do with Oil and Gas?

Dr. Luay Nakhleh, J. S. Abercrombie Professor of Computer Science at Rice University, spoke at the second Petro Community event on the importance of accounting for uncertainty when modeling complex systems like cancer growth or well performance.

Insights gained from one domain, like computational biology, can often be leveraged in other domains, like petroleum analytics, because they uncover better ways of modeling complex data to discover patterns and make more informed observations.

“Computer scientists get training in modeling and formulating problems,” states Dr. Nakhleh. “A model is an abstract representation of a system to explain some of its features, and how detailed a model is depends on the questions being asked.”

Watch more of Dr. Nakhleh’s talk to discover how he deals with modeling uncertainty in his research. Do you think this resonates with your work in oil and gas? Why or why not?

Data Science & Analytics Drilling & Completions Transfer

Demystifying Completions Data: Collecting and Organizing Data for Analytics (Part 3)

As mentioned in my previous post, in order to really be of value, we need to extend this analysis to future wells where we won’t have such a complete data set. We need to build multi-variate models using common, “always” data – like pump curves, geologic maps, or bulk data. Our approach has been for engineers to build these models directly in Spotfire through a side panel we’ve added but save these models back to a central location so that they can be version controlled and accessed by anyone in the organization. They can quickly iterate through a variety of models trained on our data set to review the model performance and sensitivity.

If we have access to historical information from previous wells, we can run our model on a variety of data sets to confirm its performance. This could be past wells that had micro seismic or where we knew there were issues with containment. Based on these diagnostics we can select a model to be applied by engineers on future developments. In order to make sure the model is used correctly we can set fences on the variables based on our training set to ensure the models are used appropriately. Because the models are built by your team – not a third-party vendor – they know exactly what assumptions and uncertainties went into the model. This approach empowers them to explore their data and answer questions without the stigma of a black-box recommendation.

Figure 1: Your team builds the models in Petro. ai – not a third-party vendor – so you know exactly what assumptions and uncertainties went into it. This approach empowers you to explore your data in new ways and answer questions without the limitations of black-box recommendations.

However, in addition to fences, we need to make sure engineers understand how and when to apply the models correctly. I won’t go into this topic much but will just say that the direction our industry is moving requires a basic level of statistics and data science understanding by all engineers and geologists, because of this has incorporated training into our standard engagements.

Slightly different hypothesis

This example used a variety of data, but it only answers one question. It’s important to note that even slight variations in the question we ask can alter what data is needed. In our example, instead of asking if a specific frac design would stay within our selected interval, we wanted to know if the vertical fracture length changed over time, we would need a different data set. Since micro seismic is a snapshot in time we wouldn’t know if the vertical frac stays open. A different data type would be needed to show these transient effects.

Data integration is often the biggest hurdle to analytics

We can start creating a map to tie back the required data needed for the questions we are interested in answering. The point of this diagram shown here is not to demonstrate the exact mapping of questions to data types, but rather, to illustrate how data integration quickly becomes a critical part of this story. This chart shows only a couple questions we may want to ask, and you can see how complicated the integration becomes. Not only are there additional questions, but new data types are constantly being added; none of which add value in isolation – there is no silver bullet, no one data type that will answer all our questions.

Figure 2: Data integration quickly becomes complicated based on the data types needed to build a robust model. There is no silver bullet. No single data type can answer all your questions.

With the pace of unconventional development, you probably don’t have time to build dedicated applications and processes for each question. You need a flexible framework to approach this analysis. Getting to an answer cannot take 6 or 12 months, by then the questions have changed and the answers are no longer relevant.

Wrap up

Bringing these data types together and analyzing them to gain cross-silo insights is critical in moving from science to scale. This is where we will find step changes in completions design and asset development that will lead to improving the capital efficiency of unconventionals. I focused on completions today, but the same story applies across the well lifecycle. Understanding what’s happening in artificial lift requires inputs from geology, drilling and completions. empowers asset teams to operationalize their data and start using it for analytics.

Three key take ways:

  • Specific questions should dictate data collection requirements.
  • Data integration is key to extracting meaningful answers.
  • We need flexible tools that can operate at the speed of unconventionals.

I’m excited about the progress we’ve already made and the direction we’re going.

Data Science & Analytics Drilling & Completions Transfer

Demystifying Completions Data: Collecting and Organizing Data for Analytics (Part 2)

As promised, let’s now walk through a specific example to illustrate an approach to analytics that we’ve seen be very effective.

I’m going to focus more on the methodology and the tools used rather than the actual analysis. The development of stacked pay is critical to the Permian as well as other plays. Containment and understanding vertical frac propagation is key to developing these resources economically. We might want to ask if a given pumping design (pump rate, intensity, landing) will stay in the target interval or break into other, less desirable rock. There are some fundamental tradeoffs that we might want to explore. For example, we may break out of zone if we pump above a given rate. If we lower the pump rate and increase the duration of the job, we need to have some confidence that the increase in day rates will yield better returns.

We can first build simulations for the frac and look at the effects of different completions designs. We can look at offset wells and historical data – though that could be challenging to piece together. We may ultimately want to validate the simulation and test different frac designs. We could do this changing the pumping schedule at different stages along the lateral of multiple wells.

Data collection

With this specific question in mind, we need to determine what data to collect. The directional survey, the formation tops (from reference well logs) and the frac van data will all be needed. However, we will also want micro seismic to see where the frac goes. Since we want to understand why the frac is either contained or not we will also need the stress profile across the intervals of interest. These could be derived from logs but ideally measured from DFITs. We may also want to collect other data types that we think could be proxies to relate back to the stress profile, like bulk seismic or interpreted geologic maps.

These data types will be collected by different vendors, at different times, and delivered to the operator in a variety of formats. We have bulk data, time series data, data processed by vendors, data interpreted by engineers and geologists. Meaningful conclusions cannot be derived from any one data type, only by integrating them can we start to see a mosaic.


Integrating the data means overcoming a series of challenges. We first need to decide where this data will live. Outlook does not make a good or sustainable data depository. Putting it all on a shared drive is not ideal as it’s difficult to relate. We could stand up a SQL database or bring all the data into an application and let it live there but both have drawbacks. Our approach leverages which uses a NoSQL back end. This provides a highly scalable and performant environment for the variety of data we will need. Also, by not trapping the data in an application (in some proprietary format) it can easily be reused to answer other questions or by other people in the future.

Getting the data co-located is a start but there’s more work to be done before we can run analytics. Throwing everything into a data lake doesn’t get us to an answer and it’s why we now have the term “data swamp”. A critical step is relating the data to each other. takes this raw data and transforms it using a standard, open data model and robust well alias system; all built from the ground up for O&G. For example, different pressure pumping vendors will have different names for common variables (maybe even different well names) that we need to reconcile. We use a well-centric data model that currently supports over 60 data types and exposes the data through an open API. also accounts for things like coordinate reference systems, time zones, and units. These are critical corrections to make since we want to be able to reuse as much of our work as possible in future analysis. Contrast this approach with the one dataset – one use case approach where you essentially rebuild the data source for every question you want to ask. We’ve seen the pitfalls of that approach as you quickly run into sustainability challenges around supporting these separate instances. At this point we have an analytics staging ground that we can actually use.

Interacting with and analyzing data

With the data integrated we need to decide how users are going to interact with the data. That could be through Matlab, Spotfire, python, excel, or PowerBI. Obviously, there are trade-offs here as well. Python and Matlab are very flexible but require a lot of user expertise. We need to consider not only the skill set of the people doing the analysis, but the skill set of the those who may ultimately leverage the insights and workflows. Do only a small group of power users need to run this analysis, or do we want every completions engineer to be able to take these results and apply them to their wells? We see a big push for the latter and so our approach has been to use a combination of custom web apps we’ve created along with O&G specific Spotfire integrations. Spotfire is widespread in O&G and it’s great for workflows. We’ve added custom visualizations and calculations to Spotfire to aid in the analysis. For example, we can bring in the directional surveys, grids, and micro seismic points to see them in 3D.

Figure 4: enables a user friendly interface, meeting engineers where they are already working with integrations to Spotfire and web apps.

We now have the data merged in an open, NoSQL back end, and have presented that processed data to end users through Spotfire where the data can be visualized and interrogated to answer our questions. We can get the well-well and well-top spacing. We can see the extent of vertical frac propagation from the micro seismic data. From here we can characterize the frac response at each stage to determine where we went out of zone. We’re building a 360 view of the reservoir to form a computational model that can be used to pull out insights.

In the third and final post of this series, we will continue this containment example and review how we can extend our analysis across an asset. We’ll also revisit the data integration challenges as we expand our approach to other questions we may want to ask while designing completions.


Data Science & Analytics Drilling & Completions Transfer

Demystifying Completions Data: Collecting and Organizing Data for Analytics (Part 1)

The oil and gas industry collects a huge amount of data trying to better understand what’s happening in the subsurface. These observations and measurements come in a range of data types that must be pieced together to garner insights. In this blog series we’ll review some of these data types and discuss an approach to integrating data to better inform decision making processes.

Before getting into the data, it’s important to note why every company needs a data strategy. Capital efficiency is now the name of the game in unconventionals. Investors are pushing for free cash flow, not just year over year increases in production. The nearby slide is from one operator but virtually every investor deck has a slide like this one. There are positive trends that operators can show – price concessions from service providers, efficiency gains in drilling, completions, facilities and increases in lateral length. Despite these gains, as an industry, shale is still not profitable. How much further can operators push these trends? How will this chart be created next year? Single-silo efficiencies are gone, and the next step change will only come from an integrated approach where the data acquired across the well lifecycle can be unlocked to fuel cross-silo insights.

Figure 1: Virtually every investor deck has a figure like this one. There are positive trends that operators can show– price concessions from service providers, efficiency gains in drilling, completions, facilities and increases in lateral length. Despite these gains, as an industry, shale is still not profitable. How much further can operators push these trends? How will this chart be created next year?

This is especially true in completions, which represent 60% of the well costs and touches so many domains. What does completions optimization mean? It’s a common phrase that gets thrown around a lot. Let’s unpack this wide-ranging topic into a series of specific questions.

  1. How does frac geometry change with completions design?
  2. How do you select an ideal landing zone?
  3. What operations sequence will lead to the best outcomes?
  4. What effect does well spacing have on production?
  5. Will diverter improve recovery?

This is just a small subset, but we can see these are complex, multidisciplinary questions. As an industry, we’re collecting and streaming massive amounts of data to try and figure this out. Companies are standing up centers of excellence around data science to get to the bottom of it. However, these issues require input from geology, geomechancis, drilling, reservoir engineering, completions, and production – the entire team. It’s very difficult to connect all the dots.

There’s also no one size fits all solution; shales are very heterogenous and your assets are very different from someone else’s, both in the subsurface and surface. Tradeoffs exist and design parameters need to be tied back to ROI. Here again, there are significant differences in strategy depending on your company’s strategy and goals.

Managing a data tsunami

When we don’t know what’s happening, we can observe, and there’s a lot of things we can observe, a lot of data we can collect. Here are some examples that I’ve grouped into two buckets: diagnostic data that you would collect specifically to better understand what’s happening and operational data that is collected as part of the job execution.

The amount of data available is massive – and only increasing as new diagnostics techniques, new acquisition systems and new edge devices come out. What data is important? What data do we really need? Collecting data is expensive so we need to make sure the value is there.

Figure 2: Here are some examples of diagnostic data that you would collect specifically to better understand what’s happening and operational data that is collected as part of the job execution.

The data we collect is of little value in isolation. Someone needs to piece everything together before we can run analytics and before we can start to see trends and insights. However, there is not standards around data formats or delivery mechanisms and so operators have had to bear the burden of stitching everything together. This is a burden not only for the operators, but also creates problems for service providers whose data is delivered as a summary pdf with raw data in Excel and is difficult to use beyond of the original job. The value of their data and their services is diminished when their work product has only limited use.

Thinking through an approach

A common approach to answering questions and collecting data is the science pad, the scope of which can vary significantly. The average unconventional well costs between $6 and 8M but a science pad can easily approach $12M and that doesn’t take into account costs of the time people will spend planning and analyzing the job. This exercise requires collecting and integrating data, applying engineering knowledge, and then building models. Taking science learnings to scale is the only way to justify the high costs associated with these projects.

Whether on a science pad or just as part of a normal completions process, data should be collected and analyzed to improve the development strategy. A scientific approach to completions optimization can help ensure continuous improvement. This starts with a hypothesis – not data collection. Start with a very specific question. This hypothesis informs what data needs to be collected. The analysis should then either validate or invalidate our hypothesis. If we end there, we’ve at least learned something, but if we can go one step further and find common or bulk data that are proxies for these diagnostics, we can scale the learnings with predictive models. Data science can play a major role here to avoid making far reaching decisions based off very few sample points. Just because we observed something in 2 or 3 wells where we collected all this data does not mean we will always see the same response. We can use data science to validate these learnings against historical data and understand the limits where we can apply them versus where we may need to collect more data.

In part 2 of this series, we’ll walk through an example of this approach that addresses vertical frac propagation. Specifically, we’ll dive into collecting, integrating, and interacting with the required data. Stay tuned!


Data Science & Analytics Transfer

Multivariate modelling using the diamond dataset

Today’s post was created by data analyst Omar Ali.

We’ll demonstrate how to create a multivariate model using the well-known diamond dataset from Kaggle. For this project, we’ll be utilizing the new Models feature in Our most recent release makes it extremely easy to run predictive algorithms on any type of dataset. This tool is constantly being upgraded with added functionality and features per our customers’ feedback. That being said, let’s predict prices!

Before we begin, please download the diamond dataset from the Kaggle page here.

Before building a model, let’s first explore the data. We want to find all the blank values or anything that has an odd distribution first to make sure we’re going to build a model on clean data.

Some quick preprocessing shows us that there are some zero values in x, y, and z. The depth and table distributions should be fenced at the distribution to keep outliers from skewing the dataset, and there are no NULL values for our cut and color samples.

The next portion of this exercise will familiarize you with the new Models feature in the

Click on the PetroPanel icon in Spotfire and select your respective database. Test your connection, and if it works, click on the “Models (beta)” tab.

Once you’re in the Models (beta) tab, you should see the following:

Click on “New Model,” name your model, and click on “Create.” This will bring you the main tab where you can edit your inputs, choose a machine learning algorithm, and save and train a model.

My Model Options tab looks like this:

To predict prices, we’ll use a random forest with 50 trees and a 20% test set hold out.

As you scroll down, you will find the Model Inputs tab. In this tab, you will point the PetroPanel to your table, select your predictors, and assign them to either “Categorical” or “Continuous.” For continuous variables, you can “fence” your dataset to keep outliers from skewing your model. For categorical variables, hot coding is the default, but a lookup table can be built as well. Here’s my Model Inputs tab:

As you can see, I fenced the depth parameter from 55 to 69 to keep the outliers from skewing the dataset.

Finally, the “Model Outputs” tab is where we can choose what we’re predicting, also with the opportunity to fence. Here’s mine:

Once everything is ready, just click “Save.” Once you save, you can go back to the “Loaded Models” tab and should see the model populated there. All you need to do now is train the model and see your results! The predicted values for your dataset will be added to the original dataset. Below you can see a visualization of the price predicted by the actual price of the diamond.

The model result metrics can be found on the Suite under the “Machine Learning” tab. The Suite is independent of the Petro Panel within Spotfire and is fully functional on its own. From here, you can view the model inputs and outputs, the variable importance, and many other metrics that are required for model evaluation. Here’s what it looks like for the model I just trained:

As you can see, we have a model with 84% accuracy on the test set. A key feature of the tool is that the model is already stored in our database, so we can create a job that runs predictions with this model at whatever frequency we decide on. This means that if we have a model that predicts diamond prices, the Suite can put it into production and the database will store the results. So, if we get diamond data daily, we can load in the data and predict the price through the Suite. If you’re looking for a one-time prediction, you can also manually enter in values for your predictions and generate a result to see how the model functions. An example of this can be found below:

I hope you enjoyed my demo of the new Multivariate Modeling feature in and the functionality of putting machine learning into production through the Suite. If you have any questions or concerns, please comment below!