Improve output file naming by using details from aztec code content (#24

)

* Add launch config 4 uic918-9 bahncard 25 from base64 raw data

* Start impl of simplified json accessor

* Try caching conan/data only

* Implement json value access with funny variable args and variant 2 distinguish objects from arrays by int and string

* Separate generic node extraction from string specific stuff

* Add ifString method 4 json access 2 provide simplified usage and forward arg pack to single place

* Add ticket key postfix 2 output file names

* Add test cases 4 bool 4 json access

* Fix optional conversion from json 2 destination value

* Add utility method 4 simplified array iteration

* Some small fixes 4 readme

* Improve docs

* Fix namespace

* Use latest botan 2.x version

* Forward error handling config flags 2 python

* Forward uic pub key file config 2 python

* Constify as much as possible args 4 decoder and analyzer

* Forward error handling config flags as ticket-decoder args

* Increase version no

* Add origin to empty interpreter results to get a better relation between input and ouput

* Improve json result creation in case interpretation fails 2 use builder

* Return json containing origin and base64 encoded raw data as a miniaml when interpretation failed or as result of file->base64

* Fix indentiation 4 minimal raw json result

* Fix python example in readme

* Disable ostream logging in case of ostream json output

* Fix typo

* Suppress logging in python context

* makeup

* Avoid some string->json->string conversions while record creation and refactor and simplify json builder stuff

* Unify and simplify json handling at some more places 2 avoid conversions

* Convert time values in u_flex records properly 2 iso time instead of raw mins

* Work around missing impl of format in gcc11 and different in gcc13 and clang14 4 durations of min

* Convert utc offset in quater hours to normal iso zone offset string 4 u_flex records

* Convert from/until date-times 2 iso instead of crazy day-offsets from issuing date-time 4 u_flex records

* Fix naming 4 date conversion method

* Support year overflow 4 date conversion from year + day-of-year

* Fix naming

.github/workflows/x86_64-macos14-clang15.yml

@@ -19,7 +19,7 @@ jobs:
       - name: Cache/Restore dependencies
         uses: actions/cache@v4
         with:
-          path: ~/.conan
+          path: ~/.conan/data
           key: ${{ runner.os }}-${{ github.workflow }}-conanfile-${{ hashFiles('conanfile.py') }}
 
       - name: Install dependencies

.github/workflows/x86_64-ubuntu22-clang14.yml

@@ -50,7 +50,7 @@ jobs:
       - name: Cache/Restore dependencies
         uses: actions/cache@v4
         with:
-          path: ~/.conan
+          path: ~/.conan/data
           key: ${{ runner.os }}-${{ github.workflow }}-conanfile-${{ hashFiles('conanfile.py') }}
 
       - name: Install dependencies

.github/workflows/x86_64-ubuntu22-gcc11.yml

@@ -46,7 +46,7 @@ jobs:
       - name: Cache/Restore dependencies
         uses: actions/cache@v4
         with:
-          path: ~/.conan
+          path: ~/.conan/data
           key: ${{ runner.os }}-${{ github.workflow }}-conanfile-${{ hashFiles('conanfile.py') }}
 
       - name: Install dependencies

.github/workflows/x86_64-ubuntu24-gcc13.yml

@@ -41,7 +41,7 @@ jobs:
       - name: Cache/Restore dependencies
         uses: actions/cache@v4
         with:
-          path: ~/.conan
+          path: ~/.conan/data
           key: ${{ runner.os }}-${{ github.workflow }}-conanfile-${{ hashFiles('conanfile.py') }}
 
       - name: Install dependencies

.gitignore

@@ -4,5 +4,6 @@
 /.venv
 /venv
 myeasylog.log
+ticketdecoder.log
 **/.DS_Store
 CMakeUserPresets.json

.vscode/launch.json

@@ -86,7 +86,7 @@
     {
       "type": "lldb",
       "request": "launch",
-      "name": "decoder - debug - images",
+      "name": "decoder - debug - images/",
       "program": "${workspaceFolder}/build/Debug/bin/ticket-decoder",
       "args": [
         "-i",
@@ -133,6 +133,18 @@
       "cwd": "${workspaceFolder}",
       "preLaunchTask": "build debug"
     },
+    {
+      "type": "lldb",
+      "request": "launch",
+      "name": "decoder - debug - base64 (Muster 918-9 BahnCard 25)",
+      "program": "${workspaceFolder}/build/Debug/bin/ticket-decoder",
+      "args": [
+        "-b",
+        "I1VUMDExMDgwMDAwMDEwLQIVAJCTCoTdr4BewUV1X5XBBpAfpSr7AhQ1ARWV8w80j6a+N0ay23jUHJvIPgAAADA1MTZ4nG1RTWsTQRgOgiVGWigF9Ti9SIJmmZn9bE9udhMTsgkhTfqBgXTWDMng7gT2I3oUlFLRg1IoFLwZQYqXSk/ixaogeCpItMcelB68+QecFT9AfC8z78Pzfjzv0+6Wi6YNEYSqjKABUR7rjrGqLmmpP4EEjCFWYBJ20S62uy3HXBNFsmE0rRYWsA4TjiwviL9hFMxy3TKbNvgnWmsNUK4CZSHTCIYuBQUy4BYJegCrIIslUPVIGNJctmBhNZdh/IYXh2xEQdOsOA2nvSS2FENQMkRT6nEA+m9eeRHrA5cysDwM8h3ukT4FlHEaAuJHHrs+oLzDHfFGLvN6JA5vUhbSsMOhLBQgFQpdinr1V6PR0AfIkCCSEsXgN+yyECBdgvgnDDVNE7Vqcg+hWpwHKnqSyHImk6nwAXFpkGU8t1gjt5hPL4NaHEY08AnnSEmmygkbawUrX499nwZAhwZSEMKKsAFrGhJ7JTSU0GCVBBHl+YoN/trT7pac4qpohDTFfZjaOJ4+OAWebdrp7JdUK5XqTc5vbE8a408remnl4uTc15P08v7H/+F3RjMX9t13W6Xnn194R5tXyuvW7kztmzNOO/np957k3J0qjU9PFd+e8R9I3HsqbJw/9Fr3Fx/vfN/eWz++/eFo7trZrSeXgur87OHsPfboJTl4vbuzN3fyA5iUuwA="
+      ],
+      "cwd": "${workspaceFolder}",
+      "preLaunchTask": "build debug"
+    },
     {
       "type": "lldb",
       "request": "launch",

.vscode/settings.json

@@ -104,7 +104,8 @@
     "strstream": "cpp",
     "charconv": "cpp",
     "execution": "cpp",
-    "format": "cpp"
+    "format": "cpp",
+    "print": "cpp"
   },
   "cmake.configureOnOpen": false,
   "terminal.integrated.env.osx": {

README.md

@@ -4,20 +4,22 @@
 [![x86_64-ubuntu24-gcc13](https://github.com/user4223/ticket-decoder/actions/workflows/x86_64-ubuntu24-gcc13.yml/badge.svg)](https://github.com/user4223/ticket-decoder/actions/workflows/x86_64-ubuntu24-gcc13.yml)
 
 # Overview
-Provide optimized and robust methods to detect and decode aztec codes by using opencv and zxing-cpp in combination and to 
-transcode UIC918 information with **signature validation** into json structure. (UIC918-3 and UIC918-9)<br>
+
+Provide optimized and robust methods to detect and decode aztec-codes by using opencv and zxing-cpp in combination and to transcode UIC918 information with **signature validation** into json structure. (UIC918-3 and UIC918-9)<br>
 
 **Looking for build instructions? Take a look at the end of this document!**
 
 ## ticket-decoder
-Decoder can detect and decode uic918 content from aztec codes from given input images/pdfs, verifies content and prints json result on stdout or dumps it into file.
+
+A command line tool to detect and decode uic918 content from aztec-codes from given input images/pdfs or raw data, verifies content and prints json result on stdout or dumps it into file.
 
 Check `ticket-decoder --help` for arguments.
 
 <img src="doc/images/decode_from_file.png" alt="ticket-decoder" height="250"/>
 <p>
 
-## ticket_decoder Python module
+## ticket_decoder (Python module)
+
 Provided python API is in an early state and the class DecoderFacade supports 2 methods right now only.
 * `decode_uic918('...')` is considered for the use case you decode the raw data from aztec-code
   in advance via zxing or other aztec-code-decoder of your choice and you want to decode
@@ -27,8 +29,7 @@ Provided python API is in an early state and the class DecoderFacade supports 2
   encode those bytes to base64 before passing it to the method.
   If your aztec-code-decoder provides a string-type only and you are able to pass
   character-encoding, try using 'ISO 8859-1' and cast the result string to raw bytes.
-* `decode_files('...')` detects and decodes aztec-codes from file or files (pdf, image) and decodes
-  raw UIC918 data to json. This is using zxing-cpp internally. It returns an array of
+* `decode_files('...')` detects and decodes aztec-codes from file or directories (pdf, image) and decodes UIC918 data to json. This is using zxing-cpp internally. It returns an array of
   tuples (input-path and json-result) of size x, while x is the amount of aztec-codes found on input.
 
 To build the module, some tools and dependencies are required. Beside python3 and essential build tools, it
@@ -53,13 +54,14 @@ See `source/python/run.py` or `source/test/python/test_decode_uic918.py` for mor
 ```
 from ticket_decoder import DecoderFacade
 
-decoder_facade = DecoderFacade()
+decoder_facade = DecoderFacade(fail_on_interpreter_error = False)
 for result in decoder_facade.decode_files('path/2/your/ticket.pdf'):
-   print(result[0] + ": " + result[1])
+   print(result[1])
 ```
 
 ## ticket-analyzer
-Analyzer is able to scan for aztec codes in images grabbed from camera or from images/pdfs in a folder. It provides a simple interactive mode to visualize detection, image processing and decoding steps and to change some parameters to find optimal setup for detection. This application is considered to optimize default parameters and algorithms for the decoder.
+
+Analyzer has a minimal UI and is able to scan for aztec-codes in images grabbed from camera or from images/pdfs in a folder. It provides a simple interactive mode to visualize detection, image processing and decoding steps and to change some parameters to find optimal setup for detection. This application is considered to optimize default parameters and algorithms for the decoder.
 
 Check `ticket-analyzer --help` for arguments.
 
@@ -71,43 +73,75 @@ To get a minimal setup for experimentation, do the following:
 * Download XML file containing public keys of issuers from https://railpublickey.uic.org/ into folder ./cert/ and name it UIC_PublicKeys.xml
 * Run `./build/Release/bin/ticket-analyzer` from workspace folder or use arguments to specify different paths to input files and folders
 * Use following keys to tweak settings:
-  * i: Visualize next image processing step
-  * I: Visualize previous image processing step
-  * c: Visualize next contour detection step
-  * C: Visualize previous contour detection step
   * f: Next image input file from image-folder
   * F: Previous image input file from image-folder
-  *  : Toggle camera device (space)
+  *  : Toggle and unpause camera device (space)
   * r: Rotate image -1 degree
   * R: Rotate image +1 degree
   * 2: Split image into 2 parts and rotate over parts
   * 4: Split image into 4 parts and rotate over parts
-  * s: Scale up image
-  * S: Scale down image
-  * 0: Reset: Rotation, Scale, Split
-  * d: Rotate over available detector implementations
-  * p: Assume pure barcode
-  * b: Use local average binarizer
-  * D: Dump current image into output-folder
-  * o: Overlay detected barcode image
-  * t: Overlay decoded content or text
+  * s: Scale image up
+  * S: Scale image down
+  * f: Flip image (x, y, x+y)
+  * 0: Reset: Rotation, Scale, Split, Flip
+  * d: Rotate over available detector implementations (forward, naive dip square detector, classifier)
+  * p: Assume pure barcode (just forwarded to zxing-cpp)
+  * b: Use local average binarizer (just forwarded to zxing-cpp)
+  * h: Try harder (just forwarded to zxing-cpp)
+  * D: Dump results into output-folder (0 disabled, 1 json, 2 json and raw data and images)
+  * g: Increase gaussian kernal for smoothing (depends on detector)
+  * i: Visualize next image processing step
+  * I: Visualize previous image processing step
+  * c: Visualize next contour detection step
+  * C: Visualize previous contour detection step
+  * o: Overlay detected barcode image (depends on detector)
+  * t: Overlay decoded content or text (depends on detector)
 * Check output-folder (./out by default) for intermediate images, raw data files or decoded data in json files
 
-## Considerations about optical Resolution
-* Printed code size: 48mm (1.89inch)
-* With 200dpi:       1.89 inch/code * 200 dot/inch ~ 380 dot/code
-* With UIC-918-3:    380  dot       / 87 blocks    ~ 4.37 dot/block
+
+# Concepts and Ideas
+
+## Design Goals
+
+* Well separated dependencies to keep it maintainable and flexible for upcoming and deprecated libraries
+  * The opencv core is an exception here since cv::Mat and other basic classes are used at all places
+* Simple environment to evaluate changes in dip chain or decoder implementations from image files, binary data as well as from camera
+
+## Modules
+
+* **I/O:**\
+Load and store for image and raw data from and to files and directories as well as read from camera device using opencv imageproc and highgui (includes PDF decoding via poppler library)
+* **Detection (DIP):**\
+Specialized [object detectors](source/lib/dip/detection/api/include/Detector.h) for barcode shapes, see section below for details
+* **Decoding:**\
+[Decoders](source/lib/barcode/api/include/Decoder.h) to get raw byte-arrays from aztec-codes (an implementation using zxing-cpp is the only one right now, but this works really good)
+* **Interpretation:**\
+[Interpreters](source/lib/uic918/api/include/Interpreter.h) to transcode different formats and records to json as well as verfication when possible (using interpreter pattern to separate the number of various, hierarchical structured formats)
+* **UI:**\
+Optional and minimal user interaction methods to support fast interactive experimentation with parameters for detection and decoding using opencv highgui (used by ticket-analyzer only)
+
+## Detector implementations
+
+* **Forward Detector:**\
+[ForwardDetector](source/lib/dip/detection/api/include/ForwardDetector.h) simply passes the entire image from pre-processor to aztec-code decoder without additional modification
+* **(Naive) DIP Square Detector:**\
+[SquareDetector](source/lib/dip/detection/api/include/SquareDetector.h) tries to detect squares by using classic image processing like smoothing, binarizing, mophological operations, convex hull, edge detection and stuff and forwards only unwarped square image parts to aztec-code decoder
+* **Classifier Detector:**\
+[ClassifierDetector](source/lib/dip/detection/api/include/ClassifierDetector.h) is prepared to use the opencv classifier to detect aztec-code objects, but there is no properly trained classifier input file right now (frontal face detection example from opencv is used as an example and for verification only)
+
 
 # Record Documentation
 
 ## U_HEAD / U_TLAY / U_FLEX
+
 * Recommendation on TAP TSI Revision - Technical Document - B12<br>
   https://www.era.europa.eu/system/files/2022-10/Recommendation%20on%20TAP%20TSI%20Revision%20-%20Technical%20Document%20-%20B12.pdf <br>
   (Link not working anymore? Go to https://www.era.europa.eu/ and search for "TAP TSI Revision - Technical Document - B12")<br>
   (TAP: Telematics Applications for Passenger Service)<br>
   (TSI: Technical Specifications for Interoperability)<br>
 
 ### Code generation for U_FLEX ASN.1 UPER
+
 * UIC-barcode
   https://github.com/UnionInternationalCheminsdeFer/UIC-barcode (Apache License 2.0)
 * Install free open source ANS.1 compiler (BSD 2)<br>
@@ -122,15 +156,18 @@ To get a minimal setup for experimentation, do the following:
 ```
 
 ## 0080VU
+
 * Interoperabilität Barcode DB Online-Ticket<br>
   https://assets.static-bahn.de/dam/jcr:8fa0c0b5-d7b8-443b-b3cd-7ae902884847/236539-315207.pdf
 
 ## 0080BL
+
 * Parser für Onlinetickets der Deutschen Bahn<br>
   https://github.com/rumpeltux/onlineticket (GPL 3.0)
 * uic-918-3<br>
   https://github.com/justusjonas74/uic-918-3 (MIT)
 
+
 # Signature Checking / Id-Mapping
 
 * Public keys from UIC<br>
@@ -142,7 +179,9 @@ To get a minimal setup for experimentation, do the following:
 * DB Railway Station Documentation (EVA-Nummern)<br>
   https://data.deutschebahn.com/dataset/data-haltestellen.html
 
+
 # Further Documentation and Ticket Samples
+
 * Interoperability UIC/VDV codes, UIC918-3 and UIC918-9 example tickets and mappings for ids used in VDV codes<br>
   https://www.bahn.de/angebot/regio/barcode
   * [UIC918-3 Muster](https://assets.static-bahn.de/dam/jcr:c362849f-210d-4dbe-bb18-34141b5ba274/mdb_320951_muster-tickets_nach_uic_918-3_2.zip)
@@ -166,19 +205,28 @@ To get a minimal setup for experimentation, do the following:
   https://community.kde.org/KDE_PIM/KItinerary/Barcode_Formats
   Some details and collection of links related to different european rail companies and their ticket formats.
 
+## Considerations about optical Resolution
+
+* Printed code size: 48mm (1.89inch)
+* With 200dpi:       1.89 inch/code * 200 dot/inch ~ 380 dot/code
+* With UIC-918-3:    380  dot       / 87 blocks    ~ 4.37 dot/block
+
 
 # Build Instructions
 
 ## Requirements
+
 * gcc >= 11, clang >= 14 (other compilers and versions may work but are not tested)
 * conan package manager < 2.0 (https://conan.io/)
 * cmake >= 3.19
 
 * python3 numpy (boost.python requires numpy for build and unfortunately, it is not possible to disable it via conan config)
 
+It is possible to build ticket-decoder and/or python module only and **to avoid the massive dependencies coming in via highgui stuff** for ticket-analyzer when it is not required. To do so, please pass `-o:h with_analyzer=False` to conan install and build the targets ticket-decoder and/or ticket_decoder via cmake only. Check [setup.Python.sh](setup.Python.sh) as a guideline.
+
 Following libraries are used by the project. Usually you should not care about it since conan will do that for you.
 
-* opencv        (image processing)
+* opencv        (image processing, image i/o and optional minimal UI)
 * zxing-cpp     (barcode/aztec-code decoding)
 * nlohmann_json (json support - output)
 * easyloggingpp (logging)
@@ -192,20 +240,23 @@ Following libraries are used by the project. Usually you should not care about i
 ## Ubuntu jammy
 
 ### Inside docker build container
+
 In general, when you want to avoid to install additional dependencies like non-default compilers and libraries on your system, consider using one of the build scripts using a docker container to create the build environment.<br>
 As long as the conanfile.py is unchanged, you can re-use the container with pre-built dependencies, source code changes are directly mirrored into build environment and artifacts are mirrored back into host system. In case dependencies change, the container gets re-build with updated dependencies.
 
-**-> this will install dependencies and run the build inside a ubuntu docker container**
+**This will install dependencies and run the build inside a ubuntu docker container**
 
-* setup.docker.ubuntu22.gcc11.sh
-* setup.docker.ubuntu22.clang15.sh
-* setup.docker.ubuntu22.gcc11.Python.sh
+* [setup.docker.ubuntu22.gcc11.sh](setup.docker.ubuntu22.gcc11.sh)
+* [setup.docker.ubuntu24.gcc13.sh](setup.docker.ubuntu24.gcc13.sh)
+* [setup.docker.ubuntu22.clang15.sh](setup.docker.ubuntu22.clang15.sh)
+* [setup.docker.ubuntu22.gcc11.Python.sh](setup.docker.ubuntu22.gcc11.Python.sh)
 
 When the preparation of the build environment has been successful, it should be possible to build the project by using `./build.sh -j` **inside the build container**.
 
 Take a look into `./build/` folder to discover artifacts. You should be able to execute the executables on host machine as well.
 
 ### On host machine
+
 When opencv has to be built from source because of missing pre-built package for your arch/os/compiler mix, it might 
 be necessary to install some further xorg/system libraries to make highgui stuff building inside conan install process. 
 To get this handled properly, use the following conan config flags:
@@ -224,7 +275,7 @@ conan profile update settings.compiler.libcxx=libstdc++11 ticket-decoder
 conan profile update conf.tools.system.package_manager:mode=install ticket-decoder
 conan profile update conf.tools.system.package_manager:sudo_askpass=True ticket-decoder
 
-git clone https://github.com/karlheinzkurt/ticket-decoder.git
+git clone https://github.com/user4223/ticket-decoder.git
 cd ticket-decoder
 ./setup.Release.sh -- -j
 
@@ -235,6 +286,7 @@ etc/python-test.sh
 ```
 
 ## MacOS with Apple clang15 (amd64 & arm64)
+
 It might be required for dependencies to get built properly during conan install to have a 
 `python` command (without 3) in path available. So when you face an error like `python: command not found`
 it might be required to create a link via `sudo ln -s $(which python3) /usr/local/bin/python` since there
@@ -247,7 +299,7 @@ pip3 install "conan<2.0" "numpy<2.0"
 conan profile new --detect --force ticket-decoder
 conan profile update settings.compiler.version=15.0 ticket-decoder
 
-git clone https://github.com/karlheinzkurt/ticket-decoder.git
+git clone https://github.com/user4223/ticket-decoder.git
 cd ticket-decoder
 ./setup.Release.sh -- -j
 

conanfile.py

@@ -3,7 +3,7 @@
 
 class TicketDecoderConan(ConanFile):
    name = 'ticket-decoder'
-   version = 'v0.9'
+   version = 'v0.10'
    settings = "os", "compiler", "build_type", "arch"
    generators = "CMakeToolchain", "CMakeDeps"
    options = {
@@ -21,7 +21,7 @@ class TicketDecoderConan(ConanFile):
                # https://conan.io/center/recipes/pugixml
                ("pugixml/1.14"),
                # https://conan.io/center/recipes/botan
-               ("botan/2.19.4"),
+               ("botan/2.19.5"),
                # https://conan.io/center/recipes/tclap
                ("tclap/1.2.5"),
                # https://conan.io/center/recipes/boost