Reducing Risk in the Petroleum Industry Machine Data and Human Intelligence Naveen Viswanath Data science Business and industry Big data architecture Get the entire collection of 50+ free data reports from O’Reilly at oreilly.com/data/free We’ve compiled the best insights from O’Reilly editors, authors, and speakers in one place, so you can dive deep into the latest of what’s happening in data FPO barcode back cover ©2016 O’Reilly Media, Inc O’Reilly is a registered trademark of O’Reilly Media, Inc D2060 Reducing Risk in the Petroleum Industry Machine Data and Human Intelligence Naveen Viswanath Beijing Boston Farnham Sebastopol Tokyo Reducing Risk in the Petroleum Industry by Naveen Viswanath Copyright © 2016 O’Reilly Media Inc All rights reserved Printed in the United States of America Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472 O’Reilly books may be purchased for educational, business, or sales promotional use Online editions are also available for most titles (http://safaribooksonline.com) For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com Editor: Tim McGovern Production Editor: Shiny Kalapurakkel Copyeditor: Gillian McGarvey August 2016: Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Rebecca Panzer First Edition Revision History for the First Edition 2016-08-11: First Release The O’Reilly logo is a registered trademark of O’Reilly Media, Inc Reducing Risk in the Petroleum Industry, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limi‐ tation responsibility for damages resulting from the use of or reliance on this work Use of the information and instructions contained in this work is at your own risk If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsi‐ bility to ensure that your use thereof complies with such licenses and/or rights 978-1-491-96471-2 [LSI] Table of Contents Reducing Risk in the Petroleum Industry: Machine Data and Human Intelligence Introduction Operational Risk Long-Term Risk Conclusion 10 Bibliography 11 v Reducing Risk in the Petroleum Industry: Machine Data and Human Intelligence Introduction To the buzzword-weary, Big Data has become the latest in the infin‐ ite series of technologies that “change the world as we know it.” But amidst the hype, there is an epochal shift: the current exponential growth in data is unprecedented and is not showing any signs of slowing down Compared to the short timelines of technology startups, the long history of the petroleum industry provides stark examples to illus‐ trate this change Seismic research happens early in the exploration and extraction stages In 1990, one square kilometer yielded 300 megabytes of seismic data In 2015, this was 10 petabytes—33 mil‐ lion times more, according to Satyam Priyadarshy, chief data scien‐ tist at Halliburton First principles, intuition, and manual arts are overwhelmed by this volume and variety of data Data-driven mod‐ els, however, can derive immense value from this data flood This report gathers highlights from Strata+Hadoop World conferences that showcase the use of data science to minimize risk in the petro‐ leum industry In the short term, data can be used to mitigate operational risk Given good data, machine learning can be used to optimize well completion parameters such as the amount and type of proppant used Ben Hamner, chief technology officer at the data science startup, Kaggle, says these are the biggest drivers of well cost and the biggest expense when drilling the well They also have a proportion‐ ate impact on how much a well can produce Using completion parameters from machine learning on one well, the gain after costs was $700,000 Priyadarshy shared how pipelining seismic, drilling, and production data can be used for long-term reservoir management Since it can be expensive to move data from offshore or remote operations, models use the data on site and the results are aggregated with pre‐ viously collected data and models Oliver Mainka (vice president of product management at SAP), Hamner, and Priyadarshy all agree that the quality of data deter‐ mines the value that can be derived from it Machines are very good at spotting new patterns in oceans of data The iterative use of human intelligence to clean the input data and validate results based on experience makes machine data-crunching an effective generator of value Big or small, using all the available data is justified if it gen‐ erates value Operational Risk The spectrum of available data can be used to answer a variety of questions High-quality input data is required for most analyses, and the output data can address different realms, like current opera‐ tional risk and longer-term organizational challenges Here are some examples of addressing operational risk during dif‐ ferent stages of the upstream process Exploration Exploration is an exciting time during which there can be immense payback for making the correct choices The right data and the information that results from this data processing can be valuable tools in the upstream arsenal Domain expertise on data sources The oil and gas industry has been a prolific user of data for a long time, as Chevron’s Martin Waterhouse points out—and just as keep‐ ing oil flowing is a complex operation running across continents, keeping information flowing can be just as much of a challenge Big oil are large companies, but they are not monoliths They are con‐ | Reducing Risk in the Petroleum Industry: Machine Data and Human Intelligence glomerations of organizations which can be considered large com‐ panies on their own The culture of the people, the role data plays, and the time that the data is retained can be very different in each organization It can take years to figure out whom to ask questions, where things are done, and how the company functions Connecting domain expertise with the latest in modeling and predictive analyt‐ ics is as important as implementing those models, but the payoff is worth it In unconventional production (shale), well production is highly cor‐ related with location Machine learning can help determine where to acquire acreage The input data can come from: Geology Core samples are rich and accurate, but also rare and very expensive Drilling and completion Amount of proppant and fluid, number of stages, and injection rate Production Publicly available in the US; varies by state Garbage in, garbage out applies here just as much as anywhere else Human intelligence is critical for quality control of data Domain experts can tell the difference between a bad sensor measurement and slowed production because of transport issues For good perfor‐ mance, a combination of manual and automated approaches is used to correct data when possible and reject otherwise Hamner esti‐ mates, 95% of the effort in tackling predictive problems in the industry lies in deeply understanding data sources and how they fit into the business use case A related challenge is how to expose results to key decision-makers Integrating disparate data sources A variety of sources can contribute to the data repository This can range from automated high-sample-rate sensors to a human drop‐ ping a rope in a tank every six months They can include audio, video, handwritten notes, and text reports The challenge is to con‐ vert these different sample rates, accuracies, accessibilities, costs, and difficulties into a validated, usable form In a case that (like many others) cuts across both data varieties and domains, Operational Risk | André Karpištšenko and his team at Marinexplore Inc (now Plan‐ etOS) have been working to ease the flow and increase the utility of ocean-related data In many parts of the world, risk is synonymous with weather The advent of inexpensive, robust drones powered by wave and solar energy has made available data that was once impossible to gather (in the eye of a storm) or too expensive (across the Pacific), which can keep us better informed of upcoming weather This can directly impact planning locations for offshore drilling platforms and ship‐ ping routes for oil tankers Risk is also equated to uncertainty In the ocean, no two days are the same and attributes like wind, waves, ocean currents, temperature, and pressure vary depending on location and time A prompt, easily accessible system is more valuable than one with long data collec‐ tion and processing times, when delays can render information use‐ less When data is democratized, the experts are not isolated anymore There are no long timelines to process and visualize data Data streams from sensors, models, and simulations are available to everyone This can even involve sharing—that often maligned word Since many data sources (satellites, models, gliders, buoys) are capital-intensive, Marinexplore started sharing public data as a dem‐ onstration of using existing resources well Now, leading companies are thinking about how to better exchange data Karpištšenko’s aim is a borderless ocean-data analysis world Drilling and Production Over the life of a well, the risk-return equation can be optimized with predictive maintenance Predictive maintenance, as understood by data folk, uses predictive analytics to understand causation and correlation with millions or even billions of records as a matter of course, and formulates predictions about machine failure in order to proactively service devices instead of relying on isolated inspections In a compressor, monitoring oil temperatures and vibrations in real time offers direct cost advantages by maximizing utility (service too soon) and minimizing downtime (service too late) by operating until the desired point on the PF curve (potential failure, functional failure) This, says Mainka, can result in big numbers Even a 0.1% reduction in maintenance costs can translate into millions of dollars | Reducing Risk in the Petroleum Industry: Machine Data and Human Intelligence saved For example, in Europe, maintenance cost is estimated to be 450 billion euros Of this, 300 billion could be addressed by mainte‐ nance improvements and 70 billon is lost due to ineffective mainte‐ nance The methods chosen for data processing should be able to handle the characteristics of the incoming data Priyadarshy highlights the characteristics of different types of upstream data During seismic studies, the volume of data is very large, but the velocity is slow and the data does not have to be analyzed in real time The value is sig‐ nificant because if you wrongly choose the drilling location of a well, it could cost you a few hundred million dollars The complementary example is during drilling The volume of data is much smaller com‐ pared to seismic studies, but the velocity is faster, and sometimes you have to analyze the data in real time If predictive models fail, it can be expensive (when a drill bit gets stuck, for example) The value of real-time data in any particular case is significant but not as high as well location Sensors in real time Sensors are becoming more pervasive, but what companies with them still varies significantly Mainka offers an example Consider six data sources, producing trillions of records Processing all of them as a matter of course, in real time, is new for 98% of compa‐ nies—even though these are sophisticated companies (Fortune 100, Fortune 500) Sensor maturity translates to lower cost and improved robustness Petabytes of data are now collected by millions of sensors The chal‐ lenge is how to use this fast enough so that value is not lost due to collection and processing Karpištšenko shares an example from the early life of Marinexplore: once buoy data was collected and ana‐ lyzed, it took a customer three months to make a decision Given that the ocean is highly dynamic, this delay seems to negate the use‐ fulness of the information Marinexplore’s platform can show meas‐ urements from sensors and data from models and simulations (such as daily sea temperatures) in seconds instead of months or years Operational Risk | Data methods A few data science methods can be applied verbatim, whereas others require tailoring to suit the petroleum industry While explaining use cases, the speakers offer a glimpse into their instantiation of this world Asset-intensive industries are especially interested in maximizing asset productivity Mainka describes how either the end user or the manufacturer is involved depending on whether the assets are owned or rented By looking at billions of records, models can create rules and back-calculate possible root causes of failure Anomalies can be either good or bad If good, try to repeat it If bad, try to avoid it Multiple rules can be chained together to classify scenarios In each case, by monitoring future performance, the system can be iteratively improved When an impending failure is detected, from the perspective of the manufacturer, the next step could be to offer preventative maintenance service for a positive customer experi‐ ence The risk of unscheduled maintenance and associated costs can thus be reduced Organizations that generate the majority of mainte‐ nance work orders from preventative and predictive inspections and use sophisticated reliability-based maintenance procedures and tools to increase asset availability have a 27% lower unplanned downtime without any increase in service and maintenance cost As with most modeling, machine learning applied to exploration and production can be validated against future performance Hamner lists the following model evaluation strategies as being use‐ ful in picking parameters for deeper study or for selecting between models: Random cross-validation Test performance with randomly withheld wells This could be biased when correlation exists between wells Time-based validation Use results from existing wells to predict new well performance This can correct for (1) but is harder in newer plays with not as many wells | Reducing Risk in the Petroleum Industry: Machine Data and Human Intelligence Spatial validation Test performance with held-out geographic areas This corrects for spatial biases and is applicable in newer plays This helps quantify acreage evaluation models In oil and gas, drilling is based on physics and first principles, with data crunching to generate metrics and evaluate key performance indicators (KPIs) However, using the volumes of data already stored, the goal is to learn, innovate, and move to holistic datadriven analytics in real time Priyadarshy details the three aspects that make now seem like the right time: Hardware From a single processor to distributed grid processing Data From local files to flexible, nonrelational distributed file systems Applications From one machine, one processor to parallel distributed frame‐ works This confluence of developments has made real-time analytics not only possible, but the new normal in industry Long-Term Risk Different aspects of long-term risk require unique approaches and solutions Practical matters whose value can be quantified, like res‐ ervoir management, are better understood than institutional ones, like loss of expertise, whose value is more difficult to quantify Practical The oil and gas industry was one of the first aggregators of large amounts of data Most of the data challenges in upstream operations revolve around storage Upstream data is expensive to gather, and it isn’t clear at the time what will be useful in the future Because com‐ panies could use it at some future time for some yet-to-bedetermined purpose, they store as much as they can Chevron has exabytes of such data, according to Waterhouse The long arc of data analytics in the industry reaches back to the ’80s and ’90s, when Chevron was an early adopter of Cray Supercomputers, used for res‐ ervoir modeling More recently, to maximize production over the Long-Term Risk | long term, reservoir characterization and reservoir simulation both use big data technologies, says Priyadarshy Institutional It is not sufficient to pick the right problem and solve it using good data It is equally important to share the results among the target population, ensure that the acquired knowledge does not perish, and future decisions are based on what was learned during a given study Any of these can be more challenging than the others, for unexpec‐ ted reasons As a model of the integration of machine learning with human expertise in materials research, Kai Trepte, lead engineer at Har‐ vard’s clean energy project, explains how building blocks are mixed in computer models and their properties analyzed The data from this analysis is mined for promising candidates, speeding up the dis‐ covery process In addition, constraints for manufacturing and dis‐ tribution are added to speed up real-world usage Combinatorially, 26 promising fragments (from research at Stanford University) resulted in 10 million molecules Help from human experimentalists and theorists, and data mining and machine learning reduced this number to 2.3 million molecules that required further study These 2.3 million molecules required 150 million calculations, generating 400 terabytes of data From that, the yield was about 0.5% The compute time for such simulations is very large, so they used an existing open source framework IBM World Computing Grid and Berkeley Open Infrastructure for Network Computing (BOINC) where volunteers donate processing on their devices With 600,000 volunteers donating 22,000 CPU years, it was equivalent to a 170,000-core supercomputer This is orders of magnitude higher than what a single, well-funded research team could afford It is dif‐ ficult to fathom how long physically making these millions of differ‐ ent molecules and testing them would take Humans and machines together made this study possible But as a general statement in research, whether academic or indus‐ trial, there is little funding for data persistence (especially when there aren’t publishable results) In short, most of the data collected during research is lost By Trepte’s estimate, within five years, 50% of raw data is lost In 10 years, 95% of data is lost This is changing | Reducing Risk in the Petroleum Industry: Machine Data and Human Intelligence The Materials Genome Initiative has funded accessibility and the infrastructure for data sharing within this field In addition to persistence of knowledge, a combination of automa‐ ted and manual approaches must be used to correct data or, if uncorrectable, to remove them from consideration Many techni‐ ques used by Hamner at Kaggle are based on expertise in datarecording and data-reporting practices, as well as experience with the types of failures that occur in the field A typical unconventional shale well may be online only for a short time during the first month It quickly spikes to peak production and then declines with most of the oil extracted within 6–12 months In Texas, public reporting of production data is at lease-level, not at well-level So there can be data corruption where the entire lease’s production is wrongly allocated to one well This can show up as spikes in production as new wells come online The risk in not cor‐ recting this is that we could wrongly deduce that this is an enor‐ mously productive well, which could throw off machine-learning models and related decisions Another problem is if production was affected because of issues not related to well potential, such as well downtime, or choked production, allocation, or or transportation The number of well data points can be small—100 to 10,000—but the cost per data point can reach $10–15 million So, this requires different data quality control than a Facebook news feed, for instance, where the number of data points is higher but the cost per data point is much lower One method that has proven to work well in well-productivity prediction is Bayesian additive regression trees (BART) This outputs not just a point estimate of the prediction but the full probability distribution that the model learned The petroleum industry doesn’t just include seismic research, drill‐ ing, mechanical maintenance, and worldwide logistics The verticals can extend all the way to the retail customer: Chevron, for example, runs many of its gas stations They make more money selling mer‐ chandise than selling gas So, optimizing this supply chain and ana‐ lyzing personal traffic and preferences has the potential for significant value One way to encourage looking at this is to form innovation zones and create places for people to play and learn This also helps peer exchanges and securing knowledge within the com‐ pany Long-Term Risk | The challenges of personal behavior aren’t limited to customer behavior analytics: they point within the company as well One chal‐ lenge is to get executives to fund longer-term initiatives For instance, it doesn’t help to only store a week’s worth of logs to study long-flow data anomalies About a year’s worth will be good Even if the current constraint is cost, understanding and solving this prob‐ lem has the potential for significant future returns Mired in quarterly financial reports during tough economic times, it might be easier to quantify savings from continuous lowest cost out‐ sourcing and offshoring But longer-term effects are much harder to quantify and may be unrecoverable Organizations can lose future data experts due to fewer opportunities for peer expertise exchange because things are done in pieces, probably at different locations In lean times, Chevron’s Waterhouse has a few ideas for data special‐ ists: • • • • • Seek alternate areas to add value Practice internally and build communities Encourage outreach Embed or relocate to interesting data roles Learn more about each business area Conclusion Analyzing a core sample of one milliliter currently can yield 100 gig‐ abytes of data If you add seismic, drilling, and other data, there are exabytes of data that need to be stored in oil fields Confronted with this, machine-data processing has huge advantages: enormous scale and processing power, no fatigue, and no cultural or other biases other than what is programmed into it But they cannot completely distinguish between good data and bad data or reasonable and unreasonable results Human intelligence is crucial to make these distinctions and make the overall system profitable 10 | Reducing Risk in the Petroleum Industry: Machine Data and Human Intelligence Bibliography Hamner, Ben “Machine Learning for Oil Exploration.” Strata + Hadoop World in San Jose 2015 February 17, 2015 Accessed August 4, 2016 http://oreil.ly/2aUHBmZ Karpištšenko, André “The Ocean’s Big Data Platform.” Strata 2014 February 11, 2014 Accessed August 4, 2016 http://oreil.ly/ 2aphxeN Mainka, Oliver “Improving Business Operations with Predictive Maintenance and Service.” Strata + Hadoop World in San Jose 2015 February 17, 2015 Accessed August 4, 2016 http://oreil.ly/ 2aEc9oj Priyadarshy, Satyam “Leveraging Big Data and Data Science in Upstream Oil and Gas Industry.” Strata + Hadoop World in San Jose 2015 February 17, 2015 Accessed August 4, 2016 http:// oreil.ly/2aRBzCz Trepte, Kai “Harvard’s Clean Energy Project: Big Data Maps to Renewable Energy.” Strata 2014 February 11, 2014 Accessed August 4, 2016 http://oreil.ly/2aUxcqq Waterhouse, Martin “Don’t Let Today’s Demands Kill Tomorrow’s Workforce!” Strata + Hadoop World in San Jose 2015 February 17, 2015 Accessed August 4, 2016 http://oreil.ly/2aRD2IK 11 About the Author Naveen Viswanath has been solving problems in the hard disk drive industry since 2000 At the intersection of data, hardware engineer‐ ing, and control software, he finds that the best challenges are inter‐ disciplinary Hailing from Chennai, India and living in Colorado, he loves mountains and the outdoors and is calmed by visits to the ocean He finds ideas plentiful during cold morning dog walks  ... Media, Inc D2060 Reducing Risk in the Petroleum Industry Machine Data and Human Intelligence Naveen Viswanath Beijing Boston Farnham Sebastopol Tokyo Reducing Risk in the Petroleum Industry by Naveen... In 10 years, 95% of data is lost This is changing | Reducing Risk in the Petroleum Industry: Machine Data and Human Intelligence The Materials Genome Initiative has funded accessibility and the. .. to make these distinctions and make the overall system profitable 10 | Reducing Risk in the Petroleum Industry: Machine Data and Human Intelligence Bibliography Hamner, Ben “Machine Learning for