Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Emit JSON simplification logs with -l SimplifyJson #516

Merged
merged 33 commits into from
Mar 21, 2024

Conversation

geo2a
Copy link
Contributor

@geo2a geo2a commented Feb 20, 2024

Summary of changes

This PR enables the -l SimplifyJson log option in kore-rpc-booster and booster-dev. This option enables structured (as JSON) logging of equation application/failures to apply an equation.

These feature is a reincarnation of the log-*-simplifications request parameter. It has become evident that bundling the logs into the response is a bad idea, as they tend to be very heavy-weight. In this iteration, we change two things:

  • logs do not contains the states anymore, but only the rule unique IDs and the failure reason (if failed)
  • logs are written to a file, rather then collected in-memory and returned in the response

Log format

the logs follow the format of the Simplification constructor of the LogEntry datatype from kore-rpc-types. This allows us to shovel both Booster's and Kore's equation application data into the same format that. Here are three examples of a log entries:

Failure in Booster: {"tag":"simplification","result":{"tag":"failure","reason":"RuleNotPreservingDefinedness","rule-id":"d0c635862a74b433473f8d45be9da2608b68fbdcf820aff3a421795196e6070b"},"origin":"booster"}
Success in Booster: [SimplifyJson] {"tag":"simplification","result":{"tag":"success","rule-id":"f310443b160236f56d095576a7500c11d1e98ee7af349b0bbf7a34b5e444d300"},"origin":"booster"}
Success in Kore: {"tag":"simplification","result":{"tag":"success","rule-id":"24f1c49206508c64e029253c37c243d97ca8b3aaea2e4d938380c89870b81955"},"origin":"kore-rpc"}

Note: we do not have a way to get the failures from Kore at that point.

Output to file or stderr

Both kore-rpc-booster and booster-dev now support a new CLI option, --log-file FILENAME. This option only has effect of -l SimplifyJson is supplied too. If both are present, the simplification logs will be written to FILENAME, otherwise they will end-up in stderr.

Note: when written to stderr, the logs are prefixed with [SimplifyJson] to distinguish them from log entries of other types. When writing to a file, this prefix is skipped.

Analysis of logs

These logs are intended for machine, rather direct human consumption. I'm working on a Python tool on top of pyk that would allow transforming these logs into a table. Something along the lines of:


|    9642 | simplification | kore-rpc | success  | BYTES-SIMPLIFICATION.bytes-concat-right-assoc-symb-l    | evm-semantics/lemmas/bytes-simplification.k:(33, 45, 33, 115)    |                              |
|    9643 | simplification | kore-rpc | success  | BYTES-SIMPLIFICATION.bytes-concat-empty-right           | evm-semantics/lemmas/bytes-simplification.k:(30, 38, 30, 65)     |                              |
|    9644 | simplification | booster  | success  |                                                         | None                                                             |                              |
|    9645 | simplification | booster  | success  |                                                         | None                                                             |                              |
|    9647 | simplification | booster  | success  |                                                         | None                                                             |                              |
|    9648 | simplification | booster  | success  | EVM.isPrecompiledAccount                                | kproj/evm-semantics/evm.md:(1340, 35, 1340, 145)                 |                              |
|    9705 | simplification | booster  | success  |                                                         | None                                                             |                              |
|   10074 | simplification | booster  | success  | EVM.isPrecompiledAccount                                | kproj/evm-semantics/evm.md:(1340, 35, 1340, 145)                 |                              |
|   10131 | simplification | booster  | success  |                                                         | None                                                             |                              |
|   10762 | simplification | kore-rpc | success  | UNKNOWN                                                 | kframework/builtin/domains.md:(1161, 8, 1161, 84)                |                              |
|   10763 | simplification | kore-rpc | success  | UNKNOWN                                                 | kframework/builtin/domains.md:(1156, 8, 1156, 55)                |                              |

the above entries as are an exert from a table produced from logs of a Kontrol proof. For proof-wide tables to be more useful, we may want to add request IDs.

@geo2a geo2a force-pushed the georgy/structured-logging branch 2 times, most recently from f704a1a to 549e9eb Compare February 23, 2024 13:06
@runtimeverification runtimeverification deleted a comment from coderabbitai bot Mar 5, 2024
rv-jenkins pushed a commit that referenced this pull request Mar 6, 2024
…#537)

This functionality is not useful, and can be implemented on top of the
JSON-logging infrastructure introduced in #516
@geo2a geo2a force-pushed the georgy/structured-logging branch 2 times, most recently from c21e088 to bf9dd42 Compare March 7, 2024 15:34
@geo2a geo2a marked this pull request as ready for review March 7, 2024 16:56
Copy link
Member

@jberthold jberthold left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Comment on lines 81 to 88
<*> optional
( strOption
( metavar "LOG_FILE"
<> long "log-file"
<> help
"Log file to write the logs to"
)
)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe not a bad idea to optionally log to a file... (but we can use a redirect, the server does not have any important output).
At the moment, this option only affects the SimplifyJSON log level, so it should maybe have a more specific name:

Suggested change
<*> optional
( strOption
( metavar "LOG_FILE"
<> long "log-file"
<> help
"Log file to write the logs to"
)
)
<*> optional
( strOption
( metavar "JSON_LOG_FILE"
<> long "simplification-log-file"
<> help
"Log file for the JSON simplification logs"
)
)

IDK if it is possible to automatically imply -l SimplifyJSON when this option is present (absolutely not required in this PR!)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've renamed --log-file to --simplification-log-file.

@rv-jenkins rv-jenkins merged commit d090886 into main Mar 21, 2024
5 checks passed
@rv-jenkins rv-jenkins deleted the georgy/structured-logging branch March 21, 2024 08:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants