Chilkat HOME Android™ Classic ASP C C++ C# Mono C# .NET Core C# C# UWP/WinRT DataFlex Delphi ActiveX Delphi DLL Visual FoxPro Java Lianja MFC Objective-C Perl PHP ActiveX PHP Extension PowerBuilder PowerShell PureBasic CkPython Chilkat2-Python Ruby SQL Server Swift 2 Swift 3/4 Tcl Unicode C Unicode C++ Visual Basic 6.0 VB.NET VB.NET UWP/WinRT VBScript Xojo Plugin Node.js Excel Go
(Chilkat2-Python) Efficiently Process a Huge XML File
Demonstrates a technique for processing a huge XML file (can be any size, even many gigabytes).
Note: This example requires Chilkat v22.214.171.124 or greater.
import sys import chilkat2 # This example shows a way to efficiently process a gigantic XML file -- one that may be too large # to fit in memory. # # Two types of XML parsers exist: DOM parsers and SAX parsers. # A DOM parser is a Document Object Model parser, where the entire XML is loaded into memory # and the application has the luxury of interacting with the XML in a convenient, random-access # way. The Chilkat Xml class is a DOM parser. Because the entire XML is loaded into memory, # huge XML files (on the order of gigabytes) are usually not loadable for memory constraints. # A SAX parser is such that the XML file is parsed as an input stream. No DOM exists. # Using a SAX parser is generally less palatable than using a DOM parser, for many reasons. # # The technique described here is a hybrid. It streams the XML file as unstructured text # to extract fragments that are individually treated as separate XML documents loaded into # the Chilkat Xml parser. # # For example, imagine your XML file is several GBs in size, but has a relatively simple structure, such as: # # <Transactions> # <Transaction id="1"> # ... # </Transaction> # <Transaction id="2"> # ... # </Transaction> # <Transaction id="3"> # ... # </Transaction> # ... # </Transactions> # In the following code, each <Transaction ...> ... </Transaction> # is extracted and loaded separately into an Xml object, where it can be manipulated # independently. The entire XML file is never entirely loaded into memory. fac = chilkat2.FileAccess() success = fac.OpenForRead("qa_data/xml/transactions.xml") if (success == False): print(fac.LastErrorText) sys.exit() xml = chilkat2.Xml() sb = chilkat2.StringBuilder() firstIteration = True retval = 1 numTransactions = 0 # The begin marker is "XML tag aware". If the begin marker begins with "<" # and ends with ">", then it is assumed to be an XML tag and it will also match # substrings where the ">" can be a whitespace char. beginMarker = "<Transaction>" endMarker = "</Transaction>" while retval == 1 : sb.Clear() # The retval can have the following values: # 0: No more fragments exist. # 1: Captured the next fragment. The text from beginMarker to endMarker, including the markers, are returned in sb. # -1: Error. retval = fac.ReadNextFragment(firstIteration,beginMarker,endMarker,"utf-8",sb) firstIteration = False if (retval == 1): numTransactions = numTransactions + 1 success = xml.LoadSb(sb,True) # Your application may now do what it needs with this particular XML fragment... if (retval < 0): print(fac.LastErrorText) print("numTransactions: " + str(numTransactions))
© 2000-2020 Chilkat Software, Inc. All Rights Reserved.