Thông tin tài liệu
www.it-ebooks.info
Implementing Splunk: Big Data
Reporting and Development for
Operational Intelligence
Learn to transform your machine data into valuable
IT and business insights with this comprehensive
and practical tutorial
Vincent Bumgarner
BIRMINGHAM - MUMBAI
www.it-ebooks.info
Implementing Splunk: Big Data Reporting and
Development for Operational Intelligence
Copyright © 2013 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval
system, or transmitted in any form or by any means, without the prior written
permission of the publisher, except in the case of brief quotations embedded in
critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy
of the information presented. However, the information contained in this book is
sold without warranty, either express or implied. Neither the author, nor Packt
Publishing, and its dealers and distributors will be held liable for any damages
caused or alleged to be caused directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the
companies and products mentioned in this book by the appropriate use of capitals.
However, Packt Publishing cannot guarantee the accuracy of this information.
First published: January 2013
Production Reference: 1140113
Published by Packt Publishing Ltd.
Livery Place
35 Livery Street
Birmingham B3 2PB, UK.
ISBN 978-1-84969-328-8
www.packtpub.com
Cover Image by Vincent Bumgarner (vincent.bumgarner@gmail.com)
www.it-ebooks.info
Credits
Author
Vincent Bumgarner
Reviewers
Mathieu Dessus
Cindy McCririe
Nick Mealy
Acquisition Editor
Kartikey Pandey
Lead Technical Editor
Azharuddin Sheikh
Technical Editors
Charmaine Pereira
Varun Pius Rodrigues
Copy Editors
Brandt D'Mello
Aditya Nair
Alda Paiva
Laxmi Subramanian
Ruta Waghmare
Project Coordinator
Anish Ramchandani
Proofreader
Martin Diver
Indexer
Tejal Soni
Graphics
Aditi Gajjar
Production Coordinator
Nitesh Thakur
Cover Work
Nitesh Thakur
www.it-ebooks.info
About the Author
Vincent Bumgarner has been designing software for nearly 20 years, working in
many languages on nearly as many platforms. He started using Splunk in 2007 and
has enjoyed watching the product evolve over the years.
While working for Splunk, he helped many companies, training dozens of users to
drive, extend, and administer this extremely exible product. At least one person at
every company he worked with asked for a book on Splunk, and he hopes his effort
helps ll their shelves.
I would like to thank my wife and kids as this book could not
have happened without their support. A big thank you to all of
the reviewers for contributing their time and expertise, and special
thanks to SplunkNinja for the recommendation.
www.it-ebooks.info
About the Reviewers
Mathieu Dessus is a security consultant for Verizon in France and acts as the
SIEM leader for EMEA. With more than 12 years of experience in the security
area, he has acquired a deep technical background in the management, design,
assessment, and systems integration of information security technologies. He
specializes in web security, Unix, SIEM, and security architecture design.
Cindy McCririe is a client architect at Splunk. In this role, she has worked with
several of Splunk's enterprise customers, ensuring successful deployment of the
technology. Many of these customers are using Splunk in unique ways. Sample
use cases include PCI compliance, security, operations management, business
intelligence, Dev/Ops, and transaction proling.
Nick Mealy was an early employee at Splunk and worked as the Mad Scientist /
Principal User Interface Developer at Splunk from March 2005 to September 2010.
He led the technical design and development of the systems that power Splunk's
search and reporting interfaces as well as on the general systems that power Splunk's
congurable views and dashboards. In 2010, he left Splunk to found his current
company, Sideview, which is creating new Splunk apps and new products on top
of the Splunk platform. The most widely known of these products is the Sideview
Utils app, which has become very widely deployed (and will be discussed in Chapter
8, Building Advanced Dashboards). Sideview Utils provides new UI modules and new
techniques that make it easier for Splunk app developers and dashboard creators to
create and maintain their custom views and dashboards.
www.it-ebooks.info
www.PacktPub.com
Support les, eBooks, discount offers and
more
You might want to visit www.PacktPub.com for support les and downloads related
to your book.
Did you know that Packt offers eBook versions of every book published, with PDF
and ePub les available? You can upgrade to the eBook version at
www.PacktPub.
com
and as a print book customer, you are entitled to a discount on the eBook copy.
Get in touch with us at service@packtpub.com for more details.
At
www.PacktPub.com, you can also read a collection of free technical articles, sign
up for a range of free newsletters and receive exclusive discounts and offers on Packt
books and eBooks.
http://PacktLib.PacktPub.com
Do you need instant solutions to your IT questions? PacktLib is Packt's online
digital book library. Here, you can access, read and search across Packt's entire
library of books.
Why Subscribe?
• Fully searchable across every book published by Packt
• Copy and paste, print and bookmark content
• On demand and accessible via web browser
Free Access for Packt account holders
If you have an account with Packt at www.PacktPub.com, you can use this to access
PacktLib today and view nine entirely free books. Simply use your login credentials
for immediate access.
www.it-ebooks.info
Table of Contents
Preface 1
Chapter 1: The Splunk Interface 7
Logging in to Splunk 7
The Home app 8
The top bar 11
Search app 13
Data generator 13
The Summary view 14
Search 16
Actions 17
Timeline 18
The eld picker 19
Fields 19
Search results 21
Options 22
Events viewer 23
Using the time picker 25
Using the eld picker 26
Using Manager 27
Summary 29
Chapter 2: Understanding Search 31
Using search terms effectively 31
Boolean and grouping operators 32
Clicking to modify your search 34
Event segmentation 34
Field widgets 34
Time 35
www.it-ebooks.info
Table of Contents
[ ii ]
Using elds to search 35
Using the eld picker 35
Using wildcards efciently 36
Only trailing wildcards are efcient 36
Wildcards are tested last 36
Supplementing wildcards in elds 37
All about time 37
How Splunk parses time 37
How Splunk stores time 37
How Splunk displays time 38
How time zones are determined and why it matters 38
Different ways to search against time 39
Specifying time in-line in your search 41
_indextime versus _time 42
Making searches faster 42
Sharing results with others 43
Saving searches for reuse 46
Creating alerts from searches 48
Schedule 49
Actions 51
Summary 52
Chapter 3: Tables, Charts, and Fields 53
About the pipe symbol 53
Using top to show common eld values 54
Controlling the output of top 56
Using stats to aggregate values 57
Using chart to turn data 61
Using timechart to show values over time 63
timechart options 65
Working with elds 66
A regular expression primer 66
Commands that create elds 68
eval 68
rex 69
Extracting loglevel 70
Using the Extract Fields interface 70
Using rex to prototype a eld 73
Using the admin interface to build a eld 75
Indexed elds versus extracted elds 77
Summary 80
www.it-ebooks.info
Table of Contents
[ iii ]
Chapter 4: Simple XML Dashboards 81
The purpose of dashboards 81
Using wizards to build dashboards 82
Scheduling the generation of dashboards 91
Editing the XML directly 91
UI Examples app 92
Building forms 92
Creating a form from a dashboard 92
Driving multiple panels from one form 97
Post-processing search results 104
Post-processing limitations 106
Panel 1 106
Panel 2 107
Panel 3 108
Final XML 108
Summary 110
Chapter 5: Advanced Search Examples 111
Using subsearches to nd loosely related events 111
Subsearch 111
Subsearch caveats 112
Nested subsearches 113
Using transaction 114
Using transaction to determine the session length 115
Calculating the aggregate of transaction statistics 117
Combining subsearches with transaction 118
Determining concurrency 122
Using transaction with concurrency 122
Using concurrency to estimate server load 123
Calculating concurrency with a by clause 124
Calculating events per slice of time 129
Using timechart 129
Calculating average requests per minute 131
Calculating average events per minute, per hour 132
Rebuilding top 134
Summary 141
Chapter 6: Extending Search 143
Using tags to simplify search 143
Using event types to categorize results 146
Using lookups to enrich data 150
Dening a lookup table le 150
www.it-ebooks.info
[...]... quick notes about the Splunk documentation: To get to documentation for search and reporting commands, quick help is provided while searching, and a link to the documentation for that command is provided through the interface When working directly with configuration files, the fastest route to the documentation for that file is to search for splunk name.conf using your favorite search engine The documentation... www.it-ebooks.info Preface Who this book is for This book should be useful for new users, seasoned users, dashboard designers, and system administrators alike This book does not try to act as a replacement for the official Splunk documentation, but should serve as a shortcut for many concepts For some sections, a good understanding of regular expressions would be helpful For some sections, the ability to read... reporting, and studying machine data This machine data usually comes from server logs, but it could also be collected from other sources Splunk is by far the most flexible and scalable solution available to tackle the huge problem of making machine data useful The goal of this book is to serve as an organized and curated guide to Splunk 4.3 As the documentation and community resources available for Splunk... Extending Splunk Writing a scripted input to gather data Capturing script output with no date Capturing script output as a single event Making a long-running scripted input Using Splunk from the command line Querying Splunk via REST Writing commands When not to write a command When to write a command Configuring commands Adding fields Manipulating data [ viii ] www.it-ebooks.info 379 379 380 382 384... custom commands, and custom actions What you need for this book To work through the examples in this book, you will need an installation of Splunk, preferably a non-production instance If you are already working with Splunk, then the concepts introduced by the examples should be applicable to your own data Splunk can be downloaded for free from http://www.splunk.com/download, for most popular platforms... Transforming data Generating data Writing a scripted lookup to enrich data Writing an event renderer Using specific fields Table of fields based on field value Pretty print XML Writing a scripted alert action to process results Summary 396 401 403 406 406 408 411 413 416 Index 417 [ ix ] www.it-ebooks.info www.it-ebooks.info Preface Splunk is a powerful tool for collecting, storing, alerting, reporting, ... Modifying metadata fields Lookup definitions Using REPORT 310 312 315 318 transforms.conf 310 [ vi ] www.it-ebooks.info Table of Contents Chaining transforms Dropping events 320 321 fields.conf 322 outputs.conf 323 indexes.conf 323 authorize.conf 325 savedsearches.conf 326 times.conf 326 commands.conf 326 web.conf 326 User interface resources 326 Views and navigation 326 Appserver resources 327 Metadata 328... more detailed manner in Chapter 7, Working with Apps Under Do more with Splunk, we find: • Add data: This links to the Add Data to Splunk page This interface is a great start for getting local data flowing into Splunk The new Preview data interface takes an enormous amount of complexity out of configuring dates and line breaking We won't go through those interfaces here, but we will go through the configuration... needed for an effective implementation of Splunk in as concise and useful a manner as possible What this book covers Chapter 1, The Splunk Interface, walks the reader through the user interface elements Chapter 2, Understanding Search, covers the basics of the search language, paying particular attention to writing efficient queries Chapter 3, Tables, Charts, and Fields, shows how to use fields for reporting, ... on objects and settings on roles • Full name and Email address are stored for the administrator's convenience • Time zone can be changed for each user This is a new feature in Splunk 4.3 Setting the time zone only affects the time zone used to display the data It is very important that the date is parsed properly when events are indexed We will discuss this in detail in Chapter 2, Understanding Search . www.it-ebooks.info Implementing Splunk: Big Data Reporting and Development for Operational Intelligence Learn to transform your machine data into valuable IT and business insights with this comprehensive and. tutorial Vincent Bumgarner BIRMINGHAM - MUMBAI www.it-ebooks.info Implementing Splunk: Big Data Reporting and Development for Operational Intelligence Copyright © 2013 Packt Publishing All rights reserved commands 390 When not to write a command 390 When to write a command 392 Conguring commands 392 Adding elds 393 Manipulating data 394 www.it-ebooks.info Table of Contents [ ix ] Transforming data
Ngày đăng: 30/03/2014, 05:20
Xem thêm: Implementing Splunk: Big Data Reporting and Development for Operational Intelligence pot, Implementing Splunk: Big Data Reporting and Development for Operational Intelligence pot, Step 5 – Mapping these apps to deployment clients in serverclass.conf