Forked from 0xdevalias/_deobfuscating-unminifying-obfuscated-web-app-code.md
Created
April 5, 2024 16:57
-
-
Save coolsoftwaretyler/d7580970d245192ad02879a2493d73c9 to your computer and use it in GitHub Desktop.
Revisions
-
0xdevalias revised this gist
Apr 5, 2024 . 1 changed file with 3 additions and 0 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -1582,6 +1582,9 @@ In addition to the links directly below, also make sure to check out the various ## Stack Graphs / Scope Graphs - See also: - https://github.com/0xdevalias/chatgpt-source-watch/issues/11 - > Explore stack graphs / scope graphs - https://github.blog/changelog/2024-03-14-precise-code-navigation-for-typescript-projects/ - > Precise code navigation is now available for all TypeScript repositories. > Precise code navigation gives more accurate results by only considering the set of classes, functions, and imported definitions that are visible at a given point in your code. -
0xdevalias revised this gist
Apr 5, 2024 . 1 changed file with 14 additions and 0 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -417,6 +417,20 @@ > - Program Dependence graphs (PDG) > - Code Property Graphs (CPG14) > - Entire graph, i.e. convert to a different graph format (ALL) - https://en.wikipedia.org/wiki/Code_property_graph - > A code property graph of a program is a graph representation of the program obtained by merging its [abstract syntax trees](https://en.wikipedia.org/wiki/Abstract_syntax_tree) (AST), [control-flow graphs](https://en.wikipedia.org/wiki/Control-flow_graph) (CFG) and [program dependence graphs](https://en.wikipedia.org/wiki/Program_dependence_graph) (PDG) at statement and predicate nodes. The resulting graph is a property graph, which is the underlying graph model of [graph databases](https://en.wikipedia.org/wiki/Graph_database) such as [Neo4j](https://en.wikipedia.org/wiki/Neo4j), [JanusGraph](https://en.wikipedia.org/wiki/JanusGraph) and [OrientDB](https://en.wikipedia.org/wiki/OrientDB) where data is stored in the nodes and edges as [key-value pairs](https://en.wikipedia.org/wiki/Key-value_pair). In effect, code property graphs can be stored in graph databases and queried using graph query languages. - > Joern CPG. The original code property graph was implemented for C/C++ in 2013 at University of Göttingen as part of the open-source code analysis tool Joern. This original version has been discontinued and superseded by the open-source Joern Project, which provides a formal code property graph specification applicable to multiple programming languages. The project provides code property graph generators for C/C++, Java, Java bytecode, Kotlin, Python, JavaScript, TypeScript, LLVM bitcode, and x86 binaries (via the Ghidra disassembler). - https://github.com/joernio/joern - > Open-source code analysis platform for C/C++/Java/Binary/Javascript/Python/Kotlin based on code property graphs. - > Joern is a platform for analyzing source code, bytecode, and binary executables. It generates code property graphs (CPGs), a graph representation of code for cross-language code analysis. Code property graphs are stored in a custom graph database. This allows code to be mined using search queries formulated in a Scala-based domain-specific query language. Joern is developed with the goal of providing a useful tool for vulnerability discovery and research in static program analysis. - https://joern.io/ - https://cpg.joern.io/ - > Code Property Graph Specification 1.1 - > This is the specification of the Code Property Graph, a language-agnostic intermediate graph representation of code designed for code querying. > > The code property graph is a directed, edge-labeled, attributed multigraph. This specification provides the graph schema, that is, the types of nodes and edges and their properties, as well as constraints that specify which source and destination nodes are permitted for each edge type. > > The graph schema is structured into multiple layers, each of which provide node, property, and edge type definitions. A layer may depend on multiple other layers and make use of the types it provides. ## Blogs / Articles / etc -
0xdevalias revised this gist
Apr 4, 2024 . 1 changed file with 62 additions and 0 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -30,6 +30,7 @@ - [`monaco-editor`](#monaco-editor) - [Obfuscation / Deobfuscation](#obfuscation--deobfuscation) - [Variable Name Mangling](#variable-name-mangling) - [Stack Graphs / Scope Graphs](#stack-graphs--scope-graphs) - [Symbolic / Concolic Execution](#symbolic--concolic-execution) - [Profiling](#profiling) - [Unsorted](#unsorted-2) @@ -1565,6 +1566,67 @@ In addition to the links directly below, also make sure to check out the various - https://github.com/estools/esshorten/blob/master/lib/utility.js#L64-L92 - `generateNextName` ## Stack Graphs / Scope Graphs - https://github.blog/changelog/2024-03-14-precise-code-navigation-for-typescript-projects/ - > Precise code navigation is now available for all TypeScript repositories. > Precise code navigation gives more accurate results by only considering the set of classes, functions, and imported definitions that are visible at a given point in your code. > > Precise code navigation is powered by the stack graphs framework. > You can read about how [we use stack graphs for code navigation](https://github.blog/2021-12-09-introducing-stack-graphs/) and visit the [stack graphs definition for TypeScript](https://github.com/github/stack-graphs/tree/main/languages/tree-sitter-stack-graphs-typescript) to learn more. - https://github.blog/2021-12-09-introducing-stack-graphs/ - > Introducing stack graphs - > Precise code navigation is powered by stack graphs, a new open source framework we’ve created that lets you define the name binding rules for a programming language using a declarative, domain-specific language (DSL). With stack graphs, we can generate code navigation data for a repository without requiring any configuration from the repository owner, and without tapping into a build process or other CI job. - LOTS of interesting stuff in this post.. - > As part of developing stack graphs, we’ve added a new graph construction language to Tree-sitter, which lets you construct arbitrary graph structures (including but not limited to stack graphs) from parsed CSTs. You use stanzas to define the gadget of graph nodes and edges that should be created for each occurrence of a Tree-sitter query, and how the newly created nodes and edges should connect to graph content that you’ve already created elsewhere. - https://github.com/tree-sitter/tree-sitter-graph - > `tree-sitter-graph` > The tree-sitter-graph library defines a DSL for constructing arbitrary graph structures from source code that has been parsed using tree-sitter. - https://marketplace.visualstudio.com/items?itemName=tree-sitter.tree-sitter-graph - > `tree-sitter-graph` support for VS Code > This language extension for VS Code provides syntax support for `tree-sitter-graph` files. - > Why aren’t we using the [Language Server Protocol](https://microsoft.github.io/language-server-protocol/) (LSP) or [Language Server Index Format](https://code.visualstudio.com/blogs/2019/02/19/lsif) (LSIF)? > > To dig even deeper and learn more, I encourage you to check out my Strange Loop talk and the `stack-graphs` crate: our open source Rust implementation of these ideas. - https://github.com/github/stack-graphs - > Stack graphs > The crates in this repository provide a Rust implementation of stack graphs, which allow you to define the name resolution rules for an arbitrary programming language in a way that is efficient, incremental, and does not need to tap into existing build or program analysis tools. - https://docs.rs/stack-graphs/latest/stack_graphs/ - https://github.com/github/stack-graphs/tree/main/languages - > This directory contains stack graphs definitions for specific languages. - https://github.com/github/stack-graphs/tree/main/languages/tree-sitter-stack-graphs-javascript - > `tree-sitter-stack-graphs` definition for JavaScript > This project defines `tree-sitter-stack-graphs` rules for JavaScript using the `tree-sitter-javascript` grammar. - > The command-line program for `tree-sitter-stack-graphs-javascript` lets you do stack graph based analysis and lookup from the command line. - `cargo install --features cli tree-sitter-stack-graphs-javascript` - `tree-sitter-stack-graphs-javascript index SOURCE_DIR` - `tree-sitter-stack-graphs-javascript status SOURCE_DIR` - `tree-sitter-stack-graphs-javascript query definition SOURCE_PATH:LINE:COLUMN` - https://github.com/github/stack-graphs/tree/main/languages/tree-sitter-stack-graphs-typescript - > `tree-sitter-stack-graphs` definition for TypeScript > This project defines `tree-sitter-stack-graphs` rules for TypeScript using the `tree-sitter-typescript` grammar. - > The command-line program for `tree-sitter-stack-graphs-typescript` lets you do stack graph based analysis and lookup from the command line. - https://dcreager.net/talks/2021-strange-loop/ - Redirects to https://dcreager.net/talks/stack-graphs/ - > Incremental, zero-config Code Navigation using stack graphs. - > In this talk I’ll describe stack graphs, which use a graphical notation to define the name binding rules for a programming language. They work equally well for dynamic languages like Python and JavaScript, and for static languages like Go and Java. Our solution is fast — processing most commits within seconds of us receiving your push. It does not require setting up a CI job, or tapping into a project-specific build process. And it is open-source, building on the tree-sitter project’s existing ecosystem of language tools. - https://www.youtube.com/watch?v=l2R1PTGcwrE - > "Incremental, zero-config Code Nav using stack graphs" by Douglas Creager - https://media.dcreager.net/dcreager-strange-loop-2021-slides.pdf - https://media.dcreager.net/dcreager-2022-ucsc-lsd-slides.pdf - https://docs.github.com/en/repositories/working-with-files/using-files/navigating-code-on-github - > GitHub has developed two code navigation approaches based on the open source [tree-sitter](https://github.com/tree-sitter/tree-sitter) and [stack-graphs](https://github.com/github/stack-graphs) library: > - Search-based - searches all definitions and references across a repository to find entities with a given name > - Precise - resolves definitions and references based on the set of classes, functions, and imported definitions at a given point in your code > > To learn more about these approaches, see "[Precise and search-based navigation](https://docs.github.com/en/repositories/working-with-files/using-files/navigating-code-on-github#precise-and-search-based-navigation)." - https://docs.github.com/en/repositories/working-with-files/using-files/navigating-code-on-github#precise-and-search-based-navigation - > Precise and search-based navigation > Certain languages supported by GitHub have access to precise code navigation, which uses an algorithm (based on the open source stack-graphs library) that resolves definitions and references based on the set of classes, functions, and imported definitions that are visible at any given point in your code. Other languages use search-based code navigation, which searches all definitions and references across a repository to find entities with a given name. Both strategies are effective at finding results and both make sure to avoid inappropriate results such as comments, but precise code navigation can give more accurate results, especially when a repository contains multiple methods or functions with the same name. - https://pl.ewi.tudelft.nl/research/projects/scope-graphs/ - > Scope Graphs | A Theory of Name Resolution - > Scope graphs provide a new approach to defining the name binding rules of programming languages. A scope graph represents the name binding facts of a program using the basic concepts of declarations and reference associated with scopes that are connected by edges. Name resolution is defined by searching for paths from references to declarations in a scope graph. Scope graph diagrams provide an illuminating visual notation for explaining the bindings in programs. ## Symbolic / Concolic Execution - https://en.wikipedia.org/wiki/Symbolic_execution -
0xdevalias revised this gist
Mar 28, 2024 . 1 changed file with 3 additions and 0 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -49,6 +49,9 @@ ## PoC - https://replit.com/@0xdevalias/Rewriting-JavaScript-Variables-via-AST-Examples - https://github.com/0xdevalias/poc-ast-tools - > poc-ast-tools > PoC scripts and tools for working with (primarily JavaScript) ASTs. ## Tools -
0xdevalias revised this gist
Jan 9, 2024 . 1 changed file with 1 addition and 1 deletion.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -997,7 +997,7 @@ - > Traverse the children of given node - https://github.com/babel/babel/blob/main/packages/babel-traverse/src/scope/index.ts#L380-L394 - ```typescript export default class Scope { uid; path: NodePath; block: t.Pattern | t.Scopable; -
0xdevalias revised this gist
Dec 24, 2023 . 1 changed file with 4 additions and 0 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -152,6 +152,10 @@ - https://www.blueclosure.com/product/bc-detect - > BC Detect (DOMinatorPro NG) helps security testers to analyse and automatically discover DOM Based Cross Site Scripting issues thanks to its Hybrid IAST Engine together with the Smart Fuzzer module. - https://www.blueclosure.com/page/features - https://github.com/kuizuo/js-deobfuscator - > JS obfuscated code restoration > Let confusion no longer be a stumbling block in reverse analysis > https://js-deobfuscator.vercel.app/ ### wakaru -
0xdevalias revised this gist
Dec 18, 2023 . 1 changed file with 1 addition and 1 deletion.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -41,7 +41,7 @@ **Other files in this gist:** - [`chrome-devtools-sources-extension.md`](#chrome-devtools-sources-extension) - > Chrome DevTools 'Sources' Extension - [`fingerprinting-minified-javascript-libraries.md`](#file-fingerprinting-minified-javascript-libraries-md) - > Fingerprinting Minified JavaScript Libraries -
0xdevalias revised this gist
Dec 18, 2023 . 1 changed file with 2 additions and 0 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -41,6 +41,8 @@ **Other files in this gist:** - [`chrome-devtools-sources-extension.md`](https://gist.github.com/0xdevalias/#chrome-devtools-sources-extension) - > Chrome DevTools 'Sources' Extension - [`fingerprinting-minified-javascript-libraries.md`](#file-fingerprinting-minified-javascript-libraries-md) - > Fingerprinting Minified JavaScript Libraries -
0xdevalias revised this gist
Dec 18, 2023 . 1 changed file with 122 additions and 0 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -0,0 +1,122 @@ ## Chrome DevTools 'Sources' Extension Originally articulated here: - https://github.com/j4k0xb/webcrack/issues/29 - > add Chrome DevTools extension that allows the web IDE to be used within DevTools - https://github.com/pionxzh/wakaru/issues/76 - > add Chrome DevTools extension that allows the web IDE to be used within DevTools ### Overview The following is as I originally wrote it on the above issues, copied here for posterity: > First off, I LOVE the new [v2.11.0](https://github.com/j4k0xb/webcrack/releases/tag/v2.11.0) update, and the changes to the web IDE to use monaco to support 'references', 'go to declaration', etc. > > This issue is about an idea I've had and wanted to work on myself for a while, but haven't got around to it yet. Often when exploring web apps, I make pretty heavy use of Chrome DevTools' ['Search'](https://developer.chrome.com/docs/devtools/search/) / ['Sources'](https://developer.chrome.com/docs/devtools/sources) / ['Network'](https://developer.chrome.com/docs/devtools/network/) / etc tabs, the [debugger](https://developer.chrome.com/docs/devtools/javascript/reference/), and the [console](https://developer.chrome.com/docs/devtools/console/api) / [utilities](https://developer.chrome.com/docs/devtools/console/utilities) APIs. While there have been some nice improvements to the 'Sources' tab over the years (pretty printing, syntax highlighting, code folding, etc); one area I have really wished it was able to support for a long time now is 'references' / 'go to definition' / similar. > > A thought I had in this space is that, while I obviously can't outright replace the 'Sources' tab (which I believe is based on [CodeMirror](https://codemirror.net/)), it should be possible to create a Chrome DevTools Extension that adds a new tab/panel that does similar to the current 'Sources' tab, but using `monaco` as it's base, and enabling features like 'references' / 'go to definition' / etc within that. > > ### Useful Chrome Extension APIs > > Overview of extending DevTools: > > - https://developer.chrome.com/docs/extensions/how-to/devtools/extend-devtools > - > Extending DevTools > > DevTools extensions add functionality to Chrome DevTools by accessing DevTools-specific extension APIs through a DevTools page added to the extension. > > Some of the Chrome Extension API's that would be useful/enable this: > > - https://developer.chrome.com/docs/extensions/reference/api/devtools/inspectedWindow > - > Use the `chrome.devtools.inspectedWindow` API to interact with the inspected window: obtain the tab ID for the inspected page, evaluate the code in the context of the inspected window, reload the page, or obtain the list of resources within the page. > - > Use the `getResources` call and the `onResourceContent` event to obtain the list of resources (documents, stylesheets, scripts, images etc) within the inspected page. The `getContent` and `setContent` methods of the `Resource` class along with the `onResourceContentCommitted` event may be used to support modification of the resource content, for example, by an external editor. > - https://developer.chrome.com/docs/extensions/reference/api/devtools/inspectedWindow#executing-code > - > The execution context of the code being evaluated includes the [Developer Tools console API](https://developers.google.com/web/tools/chrome-devtools/). For example, the code can use `inspect` and `$0`. > - https://developer.chrome.com/docs/extensions/reference/api/devtools/inspectedWindow#method-eval > - > Evaluates a JavaScript expression in the context of the main frame of the inspected page. > - This can also be used to target different `frameURL`s, different `scriptExecutionContext`s, etc > - https://developer.chrome.com/docs/extensions/reference/api/devtools/inspectedWindow#method-getResources > - > Retrieves the list of resources from the inspected page. > - https://developer.chrome.com/docs/extensions/reference/api/devtools/inspectedWindow#type-Resource > - > A resource within the inspected page, such as a document, a script, or an image. > - > `getContent`: Gets the content of the resource. (potentially encoded, Currently, only base64 is supported.). > - > `setContent`: Sets the content of the resource. Only resources with the text type are currently supported. > - https://developer.chrome.com/docs/extensions/reference/api/devtools/inspectedWindow#method-reload > - > Reloads the inspected page. > - The most interesting part of this API is probably `reloadOptions` -> `injectedScript` (though `userAgent` might be useful sometimes too): > - > `injectedScript`: If specified, the script will be injected into every frame of the inspected page immediately upon load, before any of the frame's scripts. > - > `userAgent`: If specified, the string will override the value of the User-Agent HTTP header that's sent while loading the resources of the inspected page. The string will also override the value of the `navigator.userAgent` property that's returned to any scripts that are running within the inspected page. > - https://developer.chrome.com/docs/extensions/reference/api/devtools/inspectedWindow#event-onResourceAdded > - > Fired when a new resource is added to the inspected page. > - https://developer.chrome.com/docs/extensions/reference/api/devtools/inspectedWindow#event-onResourceContentCommitted > - > Fired when a new revision of the resource is committed (e.g. user saves an edited version of the resource in the Developer Tools). > - https://developer.chrome.com/docs/extensions/reference/api/devtools/panels > - > Use the `chrome.devtools.panels` API to integrate your extension into Developer Tools window UI: create your own panels, access existing panels, and add sidebars. > - https://developer.chrome.com/docs/extensions/reference/api/devtools/panels#type-SourcesPanel > - > Represents the Sources panel > - I hadn't stumbled upon this before, but it sounds like it's possible to access the existing 'Sources' panel; can react to 'when an object is selected in the panel' (not sure what 'object' refers to in this case), can create a sidebar pane, etc > - https://developer.chrome.com/docs/extensions/reference/api/devtools/panels#method-create > - > Creates an extension panel > - This was my main original thought, in creating a new custom panel > - https://developer.chrome.com/docs/extensions/reference/api/devtools/panels#method-openResource > - > Requests DevTools to open a URL in a Developer Tools panel. > - Can seemingly be used to interact with existing panels > - https://developer.chrome.com/docs/extensions/reference/api/devtools/panels#method-setOpenResourceHandler > - > Specifies the function to be called when the user clicks a resource link in the Developer Tools window. To unset the handler, either call the method with no parameters or pass null as the parameter. > - https://developer.chrome.com/docs/extensions/reference/api/devtools/network > - Use the `chrome.devtools.network` API to retrieve the information about network requests displayed by the Developer Tools in the Network panel. > - Probably less specifically relevant to the functionality I was thinking of, but could be some value/crossover there, so figured I'd mention it. `onNavigated` / `onRequestFinished` might be useful in some cases. > - https://developer.chrome.com/docs/extensions/reference/api/devtools/recorder > - > Use the `chrome.devtools.recorder` API to customize the Recorder panel in DevTools. > > `devtools.recorder` API is a preview feature that allows you to extend the [Recorder panel](https://developer.chrome.com/docs/devtools/recorder) in Chrome DevTools. > - Again, probably not as useful, but linking here in case. > - https://developer.chrome.com/docs/extensions/reference/api/debugger > - > The `chrome.debugger` API serves as an alternate transport for Chrome's [remote debugging protocol](https://developer.chrome.com/devtools/docs/debugger-protocol). Use `chrome.debugger` to attach to one or more tabs to instrument network interaction, debug JavaScript, mutate the DOM and CSS, etc. > - > For security reasons, the `chrome.debugger` API does not provide access to all Chrome DevTools Protocol Domains. The available domains are: [Accessibility](https://chromedevtools.github.io/devtools-protocol/tot/Accessibility), [Audits](https://chromedevtools.github.io/devtools-protocol/tot/Audits), [CacheStorage](https://chromedevtools.github.io/devtools-protocol/tot/CacheStorage), [Console](https://chromedevtools.github.io/devtools-protocol/tot/Console), [CSS](https://chromedevtools.github.io/devtools-protocol/tot/css), [Database](https://chromedevtools.github.io/devtools-protocol/tot/Database), [Debugger](https://chromedevtools.github.io/devtools-protocol/tot/Debugger), [DOM](https://chromedevtools.github.io/devtools-protocol/tot/DOM), [DOMDebugger](https://chromedevtools.github.io/devtools-protocol/tot/DOMDebugger), [DOMSnapshot](https://chromedevtools.github.io/devtools-protocol/tot/DOMSnapshot), [Emulation](https://chromedevtools.github.io/devtools-protocol/tot/Emulation), [Fetch](https://chromedevtools.github.io/devtools-protocol/tot/Fetch), [IO](https://chromedevtools.github.io/devtools-protocol/tot/IO), [Input](https://chromedevtools.github.io/devtools-protocol/tot/Input), [Inspector](https://chromedevtools.github.io/devtools-protocol/tot/Inspector), [Log](https://chromedevtools.github.io/devtools-protocol/tot/Log), [Network](https://chromedevtools.github.io/devtools-protocol/tot/Network), [Overlay](https://chromedevtools.github.io/devtools-protocol/tot/Overlay), [Page](https://chromedevtools.github.io/devtools-protocol/tot/Page), [Performance](https://chromedevtools.github.io/devtools-protocol/tot/Performance), [Profiler](https://chromedevtools.github.io/devtools-protocol/tot/Profiler), [Runtime](https://chromedevtools.github.io/devtools-protocol/tot/Runtime), [Storage](https://chromedevtools.github.io/devtools-protocol/tot/Storage), [Target](https://chromedevtools.github.io/devtools-protocol/tot/Target), [Tracing](https://chromedevtools.github.io/devtools-protocol/tot/Tracing), [WebAudio](https://chromedevtools.github.io/devtools-protocol/tot/WebAudio), and [WebAuthn](https://chromedevtools.github.io/devtools-protocol/tot/WebAuthn). > - This can do all sorts of powerful stuff, though some of the original reasons I was thinking of needing to use it (eg. getting access to the page's script contents/etc) are apparently already covered in other more specific APIs (eg. `chrome.devtools.inspectedWindow`). Still worth being aware of it and the features it has though, as there is some pretty cool stuff here! > - https://chromedevtools.github.io/devtools-protocol/ > > Then there are also all of the 'standard' Chrome Extension APIs as well, which can do a lot of cool stuff: > > - https://developer.chrome.com/docs/extensions/reference/api > > A few of which could be useful for this feature: > > - https://developer.chrome.com/docs/extensions/reference/api/runtime > - > Use the `chrome.runtime` API to retrieve the service worker, return details about the manifest, and listen for and respond to events in the app or extension lifecycle. You can also use this API to convert the relative path of URLs to fully-qualified URLs. > - Some more esotetic, but interesting aspects of this, are the ability to communicate with other externsions and/or locally installed programs using `connect` / `onConnectExternal` / `onMessageExternal` / `connectNative` / `sendMessageNative`: > - https://developer.chrome.com/docs/extensions/reference/api/runtime#method-connect > - > Attempts to connect listeners within an extension/app (such as the background page), or other extensions/apps. This is useful for content scripts connecting to their extension processes, inter-app/extension communication, and [web messaging](https://developer.chrome.com/docs/extensions/manifest/externally_connectable). Note that this does not connect to any listeners in a content script. Extensions may connect to content scripts embedded in tabs via [`tabs.connect`](https://developer.chrome.com/docs/extensions/reference/tabs/#method-connect). > - https://developer.chrome.com/docs/extensions/reference/api/runtime#event-onConnectExternal > - > Fired when a connection is made from another extension (by [`runtime.connect`](https://developer.chrome.com/docs/extensions/reference/api/runtime#method-connect)). > - https://developer.chrome.com/docs/extensions/reference/api/runtime#event-onMessageExternal > - > Fired when a message is sent from another extension/app (by [`runtime.sendMessage`](https://developer.chrome.com/docs/extensions/reference/api/runtime#method-sendMessage)). > - https://developer.chrome.com/docs/extensions/reference/api/runtime#method-connectNative > - > Connects to a native application in the host machine. See [Native Messaging](https://developer.chrome.com/docs/extensions/nativeMessaging) for more information. > - https://developer.chrome.com/docs/extensions/reference/api/runtime#method-sendNativeMessage > - > Send a single message to a native application. > - https://developer.chrome.com/docs/extensions/reference/api/runtime#event-onConnectNative > - > Fired when a connection is made from a native application. Currently only supported on Chrome OS. > - While this is only supported from Chrome OS, I believe the browser can always initiate the connection to a native app. > - https://developer.chrome.com/docs/extensions/reference/api/permissions > - > Use the `chrome.permissions` API to request [declared optional permissions](https://developer.chrome.com/docs/extensions/mv3/declare_permissions/) at run time rather than install time, so users understand why the permissions are needed and grant only those that are necessary. > - https://developer.chrome.com/docs/extensions/reference/api/contextMenus > - > Use the `chrome.contextMenus` API to add items to Google Chrome's context menu. You can choose what types of objects your context menu additions apply to, such as images, hyperlinks, and pages. > - https://developer.chrome.com/docs/extensions/reference/api/sidePanel > - > Use the `chrome.sidePanel` API to host content in the browser's side panel alongside the main content of a webpage. > - https://developer.chrome.com/docs/extensions/reference/api/storage > - > Use the `chrome.storage` API to store, retrieve, and track changes to user data. > > The Storage API provides an extension-specific way to persist user data and state. It's similar to the web platform's storage APIs ([IndexedDB](https://developer.mozilla.org/docs/Web/API/Window/indexeddb), and [Storage](https://developer.mozilla.org/docs/Web/API/Storage)), but was designed to meet the storage needs of extensions. > - https://developer.chrome.com/docs/extensions/reference/api/notifications > - Use the `chrome.notifications` API to create rich notifications using templates and show these notifications to users in the system tray. > > And some that are a little more esoteric, but might still be interesting/useful: > > - https://developer.chrome.com/docs/extensions/reference/api/scripting > - > Use the `chrome.scripting` API to execute script in different contexts. > - https://developer.chrome.com/docs/extensions/reference/api/userScripts > - > Use the `userScripts` API to execute user scripts in the User Scripts context. > - https://developer.chrome.com/docs/extensions/reference/api/webNavigation > - > Use the `chrome.webNavigation` API to receive notifications about the status of navigation requests in-flight. > - https://developer.chrome.com/docs/extensions/reference/api/offscreen > - > Use the `offscreen` API to create and manage offscreen documents. -
0xdevalias revised this gist
Dec 10, 2023 . 1 changed file with 2 additions and 1 deletion.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -1716,4 +1716,5 @@ These are private chat links, so won't work for others, and are included here on - [Reverse engineering ChatGPT's frontend web app + deep dive explorations of the code (0xdevalias gist)](https://gist.github.com/0xdevalias/4ac297ee3f794c17d0997b4673a2f160#reverse-engineering-chatgpts-frontend-web-app--deep-dive-explorations-of-the-code) - [Bypassing Cloudflare, Akamai, etc (0xdevalias gist)](https://gist.github.com/0xdevalias/b34feb567bd50b37161293694066dd53#bypassing-cloudflare-akamai-etc) - [devalias' Beeper CSS Hacks (0xdevalias gist)](https://gist.github.com/0xdevalias/3d2f5a861335cc1277b21a29d1285cfe#beeper-custom-theme-styles) - [Reverse Engineering Golang (0xdevalias' gist)](https://gist.github.com/0xdevalias/4e430914124c3fd2c51cb7ac2801acba#reverse-engineering-golang) - [Reverse Engineering on macOS (0xdevalias' gist)](https://gist.github.com/0xdevalias/256a8018473839695e8684e37da92c25#reverse-engineering-on-macos) -
0xdevalias revised this gist
Dec 7, 2023 . 1 changed file with 5 additions and 0 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -4,6 +4,7 @@ <!-- TOC start (generated with https://derlin.github.io/bitdowntoc/) --> - [PoC](#poc) - [Tools](#tools) - [Unsorted](#unsorted) - [wakaru](#wakaru) @@ -43,6 +44,10 @@ - [`fingerprinting-minified-javascript-libraries.md`](#file-fingerprinting-minified-javascript-libraries-md) - > Fingerprinting Minified JavaScript Libraries ## PoC - https://replit.com/@0xdevalias/Rewriting-JavaScript-Variables-via-AST-Examples ## Tools ### Unsorted -
0xdevalias revised this gist
Dec 6, 2023 . 1 changed file with 437 additions and 2 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -1,9 +1,23 @@ # Fingerprinting Minified JavaScript Libraries ## Table of Contents <!-- TOC start (generated with https://derlin.github.io/bitdowntoc/) --> - [See Also](#see-also) - [Initial ChatGPT Conversation / Notes](#initial-chatgpt-conversation--notes) - [Thoughts / comments as I've articulated them elsewhere](#thoughts--comments-as-ive-articulated-them-elsewhere) - [On `j4k0xb/webcrack`](#on-j4k0xbwebcrack) - [Issue 21: rename short identifiers](#issue-21-rename-short-identifiers) - [On `pionxzh/wakaru`](#on-pionxzhwakaru) - [Issue 34: support `un-mangle` identifiers](#issue-34-support-un-mangle-identifiers) <!-- TOC end --> ## See Also - https://github.com/pionxzh/wakaru/issues/41 - > Module detection - https://github.com/pionxzh/wakaru/issues/34 - > support `un-mangle` identifiers ## Initial ChatGPT Conversation / Notes @@ -117,4 +131,425 @@ Source: https://chat.openai.com/c/d9b7b64f-aa93-474e-939f-79e376e6d375 > > 1. **Respecting Copyright**: Ensure that the use of library fingerprints complies with legal and ethical standards, respecting the intellectual property rights of library authors. > > This project aligns well with your background in IT and software engineering. The main challenge lies in the nuances of JavaScript minification and the diverse coding patterns across different libraries, but a focused and iterative approach can lead to a robust and useful tool for the community. ## Thoughts / comments as I've articulated them elsewhere ### On `j4k0xb/webcrack` #### Issue 21: rename short identifiers > When I was exploring this concept in my own deobfuscation PoC project, I was exploring to make the variable names unique + have them add sort of semantic information about their source/scope. > > Eg. if it was an arg to a function, it might be `arg_1`. Or potentially if the function is `foo`, it might end up as `foo_arg_1` > > It looks like most of the PoC code I was playing with was local/in a pretty messy/hacky state, but I did find a link in it to an online REPL I was playing around with some of it in. Not sure how outdated that code is, but it might be useful: > > - https://replit.com/@0xdevalias/Rewriting-JavaScript-Variables-via-AST-Examples > > There were a number of different AST parsers I was playing around with, but I think that this babel code may have been the latest (not sure which one): > > - https://replit.com/@0xdevalias/Rewriting-JavaScript-Variables-via-AST-Examples#babel_v1_3.js > - https://replit.com/@0xdevalias/Rewriting-JavaScript-Variables-via-AST-Examples#babel_v1_3_clean.js > > Within those files, I believe the functions `getNameFromPath`, `getPrefix` (and older commented out functions `getTypePrefix`, `getPrefix` > > --- > > **Edit:** Came across this in another issue here: > > > I published my decompiler that I used in the above example. I think it might be a good reference for adding this feature. > > https://github.com/e9x/krunker-decompiler > > > > _Originally posted by @e9x in https://github.com/j4k0xb/webcrack/issues/10#issuecomment-1546764062_ > > And looking at it's `libRenameVars` code seems to be taking a vaguely similar approach to how I was looking at doing things in my original PoC that I described above: > > - https://github.com/e9x/krunker-decompiler/blob/master/src/libRenameVars.ts > - `getVarPrefix` will set a prefix based on the type (eg. `func`, `arg`, `Class`, `imported`, `var`) > - `getName` generates a new variable name that does not conflict with existing names or reserved keywords > - `generateName` generates a new name for a variable considering its scope, type, and the context in which it is used (e.g., whether it's a class, a function variable, etc.). > It employs various AST manipulations to ensure the generated name is appropriate and does not conflict with existing names. > > A more generalised summary/overview (via ChatGPT): > > > Certainly, the code implements a sophisticated algorithm for renaming variables in a JavaScript program, adhering to several high-level rules and strategies: > > > > 1. **Type-Specific Prefixing**: > > - The `getVarPrefix` function assigns specific prefixes to variable names based on their type (e.g., "func" for function names, "arg" for parameters). This approach helps in identifying the role of a variable just by its name. > > > > 2. **Avoiding Reserved Keywords**: > > - The script includes a comprehensive list of reserved JavaScript keywords. If a variable's name matches a reserved keyword, it is prefixed with an underscore to prevent syntax errors. > > > > 3. **Unique Naming with Context Consideration**: > > - The `generateName` function ensures that each variable gets a unique name that doesn't conflict with other variables in its scope. It also considers the context in which a variable is used. For example, if a variable is part of a class, it may receive a name that reflects this context, using `pascalCase` or `camelCase` as appropriate. > > > > 4. **Handling Special Cases**: > > - The script contains logic to handle special cases, such as variables that are function expressions (`isFuncVar`) or class instances (`isClass`). This affects the naming convention applied to these variables. > > > > 5. **Randomness with Mersenne Twister**: > > - A Mersenne Twister is used to generate random elements for variable names, ensuring that the names are not only unique within the scope of the program but also less predictable. > > > > 6. **AST-Based Renaming**: > > - The script analyzes the Abstract Syntax Tree (AST) of the program to understand the structure and scope of variables. This analysis guides the renaming process, ensuring that the new names are consistent with the variable's usage and position in the code. > > > > 7. **Scope Analysis with ESLint Scope**: > > - By leveraging `eslint-scope`, the script can accurately determine the scope of each variable. This is crucial in avoiding name collisions and ensuring that the renaming respects lexical scoping rules in JavaScript. > > > > 8. **Consideration for Exported and Assigned Variables**: > > - The script pays special attention to variables that are exported or assigned in specific ways (e.g., through `Object.defineProperty`). It ensures that these variables receive names that are appropriate for their roles. > > > > In summary, the script uses a combination of type-based naming conventions, context consideration, randomness, AST analysis, and scope analysis to systematically rename variables in a JavaScript program. This approach aims to enhance readability, avoid conflicts, and maintain the logical structure of the program. > > _Originally posted by @0xdevalias in https://github.com/j4k0xb/webcrack/issues/21#issuecomment-1807290502_ --- > And for an even cooler/more extreme version of improving variable naming; I just came across this blog post / project from @jehna that makes use of `webcrack` + ChatGPT for variable renaming: > > - https://thejunkland.com/blog/using-llms-to-reverse-javascript-minification.html > - > Using LLMs to reverse JavaScript variable name minification > > This blog introduces a novel way to reverse minified Javascript using large language models (LLMs) like ChatGPT and llama2 while keeping the code semantically intact. The code is open source and available at [Github project Humanify](https://github.com/jehna/humanify). > - https://github.com/jehna/humanify > - > Un-minify Javascript code using ChatGPT > - > This tool uses large language modeles (like ChatGPT & llama2) and other tools to un-minify Javascript code. Note that LLMs don't perform any structural changes – they only provide hints to rename variables and functions. The heavy lifting is done by Babel on AST level to ensure code stays 1-1 equivalent. > > _Originally posted by @0xdevalias in https://github.com/j4k0xb/webcrack/issues/21#issuecomment-1807342986_ --- > I came across another tool today that seemed to have a start on implementing some 'smart rename' features: > > - https://github.com/pionxzh/wakaru > - https://wakaru.vercel.app/ > > Digging through the code lead me to this: > > - https://github.com/pionxzh/wakaru/tree/main/packages/unminify#smart-rename > - > Rename minified identifiers with heuristic rules. > - https://github.com/pionxzh/wakaru/blob/main/packages/unminify/src/transformations/smart-rename.ts > - `handleDestructuringRename`, `handleFunctionParamsRename`, `handlePropertyRename`, `handleReactRename`, `getElementName` > - https://github.com/pionxzh/wakaru/blob/main/packages/unminify/src/utils/identifier.ts#L28-L75 > - `generateName`, `getUniqueName` > - https://github.com/pionxzh/wakaru/blob/master/packages/unminify/src/transformations/__tests__/smart-rename.spec.ts > > There's also an issue there that seems to be exploring how to improve 'unmangling variable names' as well: > > - https://github.com/pionxzh/wakaru/issues/34 > > Which I wrote the following extra thoughts on: > > > I just finished up writing some thoughts/references for variable renaming on the `webcrack` repo, that could also be a useful idea for here. (see quotes below) > > > > When I was exploring PoC ideas for my own project previously, I was looking to generate a file similar to the 'module map' that this project is using; but instead of just for the names of modules, I wanted to be able to use it to provide a 'variable name map'. Though because the specific variables used in webpack/etc can change between builds, my thought was that first 'normalising' them to a 'known format' based on their context would make sense to do first. > > > > That could then be letter enhanced/expanded by being able to pre-process these 'variable name mappings' for various open source projects in a way that could then be applied 'automagically' without the end user needing to first create them. > > > > It could also be enhanced by similar techniques such as what the `humanify` project does, by using LLMs/similar to generate suggested variable name mappings based on the code. > > > > My personal ideal end goal for a feature like that would then allow me to use it within an IDE-like environment, where I can rename variables 'as I explore', knowing that the mappings/etc will be kept up to date. > > > > _Originally posted by @0xdevalias in https://github.com/pionxzh/wakaru/issues/34#issuecomment-1807393509_ > > _Originally posted by @0xdevalias in https://github.com/j4k0xb/webcrack/issues/21#issuecomment-1807398430_ --- > Another link from my reference notes that I forgot to include earlier; my thoughts on how to rename otherwise unknown variables are based on similar concepts that are used in reverse engineering tools such as IDA: > > - https://hex-rays.com/blog/igors-tip-of-the-week-34-dummy-names/ > - > In IDA’s disassembly, you may have often observed names that may look strange and cryptic on first sight: `sub_73906D75`, `loc_40721B`, `off_40A27C` and more. In IDA’s terminology, they’re called dummy names. They are used when a name is required by the assembly syntax but there is nothing suitable available > - https://www.hex-rays.com/products/ida/support/idadoc/609.shtml > - > IDA Help: Names Representation > - > Dummy names are automatically generated by IDA. They are used to denote subroutines, program locations and data. Dummy names have various prefixes depending on the item type and value > > --- > > > And a few more I was looking at recently as well (that is sort of basically `smart-rename`: > > > > - https://binary.ninja/2023/09/15/3.5-expanded-universe.html#automatic-variable-naming > > - > Automatic Variable Naming > > > One easy way to improve decompilation output is to come up with [better default names](https://github.com/Vector35/binaryninja-api/issues/2558) for variables. There’s a lot of possible defaults you could choose and a number of different strategies are seen throughout different reverse engineering tools. Prior to 3.5, Binary Ninja left variables named based on their origin. Stack variables were var_OFFSET, register-based variables were reg_COUNTER, and global data variables were (data_). While this scheme isn’t changing, we’re being much more intelligent about situations where additional information is available. > > > > > > For example, if a variable is passed to a function and a variable name is available, we can now make a much better guess for the variable name. This is most obvious in binaries with type libraries. > > - > This isn’t the only style of default names. Binary Ninja also will name loop counters with simpler names like `i`, or `j`, `k`, etc (in the case of nested loops) > > - https://github.com/Vector35/binaryninja-api/issues/2558 > > > > _Originally posted by @0xdevalias in https://github.com/pionxzh/wakaru/issues/34#issuecomment-1822263687_ > > _Originally posted by @0xdevalias in https://github.com/j4k0xb/webcrack/issues/21#issuecomment-1822262649_ ### On `pionxzh/wakaru` #### Issue 34: support `un-mangle` identifiers > I just finished up writing some thoughts/references for variable renaming on the `webcrack` repo, that could also be a useful idea for here. (see quotes below) > > When I was exploring PoC ideas for my own project previously, I was looking to generate a file similar to the 'module map' that this project is using; but instead of just for the names of modules, I wanted to be able to use it to provide a 'variable name map'. Though because the specific variables used in webpack/etc can change between builds, my thought was that first 'normalising' them to a 'known format' based on their context would make sense to do first. > > That could then be later enhanced/expanded by being able to pre-process these 'variable name mappings' for various open source projects in a way that could then be applied 'automagically' without the end user needing to first create them. > > It could also be enhanced by similar techniques such as what the `humanify` project does, by using LLMs/similar to generate suggested variable name mappings based on the code. > > My personal ideal end goal for a feature like that would then allow me to use it within an IDE-like environment, where I can rename variables 'as I explore', knowing that the mappings/etc will be kept up to date. > > --- > > > When I was exploring this concept in my own deobfuscation PoC project, I was exploring to make the variable names unique + have them add sort of semantic information about their source/scope. > > > > Eg. if it was an arg to a function, it might be `arg_1`. Or potentially if the function is `foo`, it might end up as `foo_arg_1` > > > > It looks like most of the PoC code I was playing with was local/in a pretty messy/hacky state, but I did find a link in it to an online REPL I was playing around with some of it in. Not sure how outdated that code is, but it might be useful: > > > > - https://replit.com/@0xdevalias/Rewriting-JavaScript-Variables-via-AST-Examples > > > > There were a number of different AST parsers I was playing around with, but I think that this babel code may have been the latest (not sure which one): > > > > - https://replit.com/@0xdevalias/Rewriting-JavaScript-Variables-via-AST-Examples#babel_v1_3.js > > - https://replit.com/@0xdevalias/Rewriting-JavaScript-Variables-via-AST-Examples#babel_v1_3_clean.js > > > > Within those files, I believe the functions `getNameFromPath`, `getPrefix` (and older commented out functions `getTypePrefix`, `getPrefix` > > > > --- > > > > **Edit:** Came across this in another issue here: > > > > > I published my decompiler that I used in the above example. I think it might be a good reference for adding this feature. > > > https://github.com/e9x/krunker-decompiler > > > > > > _Originally posted by @e9x in https://github.com/j4k0xb/webcrack/issues/10#issuecomment-1546764062_ > > > > And looking at it's `libRenameVars` code seems to be taking a vaguely similar approach to how I was looking at doing things in my original PoC that I described above: > > > > - https://github.com/e9x/krunker-decompiler/blob/master/src/libRenameVars.ts > > - `getVarPrefix` will set a prefix based on the type (eg. `func`, `arg`, `Class`, `imported`, `var`) > > - `getName` generates a new variable name that does not conflict with existing names or reserved keywords > > - `generateName` generates a new name for a variable considering its scope, type, and the context in which it is used (e.g., whether it's a class, a function variable, etc.). > > It employs various AST manipulations to ensure the generated name is appropriate and does not conflict with existing names. > > > > A more generalised summary/overview (via ChatGPT): > > > > > Certainly, the code implements a sophisticated algorithm for renaming variables in a JavaScript program, adhering to several high-level rules and strategies: > > > > > > 1. **Type-Specific Prefixing**: > > > - The `getVarPrefix` function assigns specific prefixes to variable names based on their type (e.g., "func" for function names, "arg" for parameters). This approach helps in identifying the role of a variable just by its name. > > > > > > 2. **Avoiding Reserved Keywords**: > > > - The script includes a comprehensive list of reserved JavaScript keywords. If a variable's name matches a reserved keyword, it is prefixed with an underscore to prevent syntax errors. > > > > > > 3. **Unique Naming with Context Consideration**: > > > - The `generateName` function ensures that each variable gets a unique name that doesn't conflict with other variables in its scope. It also considers the context in which a variable is used. For example, if a variable is part of a class, it may receive a name that reflects this context, using `pascalCase` or `camelCase` as appropriate. > > > > > > 4. **Handling Special Cases**: > > > - The script contains logic to handle special cases, such as variables that are function expressions (`isFuncVar`) or class instances (`isClass`). This affects the naming convention applied to these variables. > > > > > > 5. **Randomness with Mersenne Twister**: > > > - A Mersenne Twister is used to generate random elements for variable names, ensuring that the names are not only unique within the scope of the program but also less predictable. > > > > > > 6. **AST-Based Renaming**: > > > - The script analyzes the Abstract Syntax Tree (AST) of the program to understand the structure and scope of variables. This analysis guides the renaming process, ensuring that the new names are consistent with the variable's usage and position in the code. > > > > > > 7. **Scope Analysis with ESLint Scope**: > > > - By leveraging `eslint-scope`, the script can accurately determine the scope of each variable. This is crucial in avoiding name collisions and ensuring that the renaming respects lexical scoping rules in JavaScript. > > > > > > 8. **Consideration for Exported and Assigned Variables**: > > > - The script pays special attention to variables that are exported or assigned in specific ways (e.g., through `Object.defineProperty`). It ensures that these variables receive names that are appropriate for their roles. > > > > > > In summary, the script uses a combination of type-based naming conventions, context consideration, randomness, AST analysis, and scope analysis to systematically rename variables in a JavaScript program. This approach aims to enhance readability, avoid conflicts, and maintain the logical structure of the program. > > > > _Originally posted by @0xdevalias in https://github.com/j4k0xb/webcrack/issues/21#issuecomment-1807290502_ > > --- > > > And for an even cooler/more extreme version of improving variable naming; I just came across this blog post / project from @jehna that makes use of `webcrack` + ChatGPT for variable renaming: > > > > - https://thejunkland.com/blog/using-llms-to-reverse-javascript-minification.html > > - > Using LLMs to reverse JavaScript variable name minification > > > This blog introduces a novel way to reverse minified Javascript using large language models (LLMs) like ChatGPT and llama2 while keeping the code semantically intact. The code is open source and available at [Github project Humanify](https://github.com/jehna/humanify). > > - https://github.com/jehna/humanify > > - > Un-minify Javascript code using ChatGPT > > - > This tool uses large language modeles (like ChatGPT & llama2) and other tools to un-minify Javascript code. Note that LLMs don't perform any structural changes – they only provide hints to rename variables and functions. The heavy lifting is done by Babel on AST level to ensure code stays 1-1 equivalent. > > > > _Originally posted by @0xdevalias in https://github.com/j4k0xb/webcrack/issues/21#issuecomment-1807342986_ > > _Originally posted by @0xdevalias in https://github.com/pionxzh/wakaru/issues/34#issuecomment-1807393509_ > > For now, we have `smart-rename` that can guess the variable name based on the context. I would like to expand it to cover some other generic cases. > > Linking to my `smart-rename` related issues to keep the contextual link here: > > - https://github.com/pionxzh/wakaru/issues/40 > - https://github.com/pionxzh/wakaru/issues/48 > - https://github.com/pionxzh/wakaru/issues/49 > > _Originally posted by @0xdevalias in https://github.com/pionxzh/wakaru/issues/34#issuecomment-1818578969_ --- > > Another link from my reference notes that I forgot to include earlier; my thoughts on how to rename otherwise unknown variables are based on similar concepts that are used in reverse engineering tools such as IDA: > > > > - https://hex-rays.com/blog/igors-tip-of-the-week-34-dummy-names/ > > - > In IDA’s disassembly, you may have often observed names that may look strange and cryptic on first sight: `sub_73906D75`, `loc_40721B`, `off_40A27C` and more. In IDA’s terminology, they’re called dummy names. They are used when a name is required by the assembly syntax but there is nothing suitable available > > - https://www.hex-rays.com/products/ida/support/idadoc/609.shtml > > - > IDA Help: Names Representation > > - > Dummy names are automatically generated by IDA. They are used to denote subroutines, program locations and data. Dummy names have various prefixes depending on the item type and value > > > > _Originally posted by @0xdevalias in https://github.com/j4k0xb/webcrack/issues/21#issuecomment-1822262649_ > > --- > > And a few more I was looking at recently as well (that is sort of basically `smart-rename`: > > - https://binary.ninja/2023/09/15/3.5-expanded-universe.html#automatic-variable-naming > - > Automatic Variable Naming > > One easy way to improve decompilation output is to come up with [better default names](https://github.com/Vector35/binaryninja-api/issues/2558) for variables. There’s a lot of possible defaults you could choose and a number of different strategies are seen throughout different reverse engineering tools. Prior to 3.5, Binary Ninja left variables named based on their origin. Stack variables were var_OFFSET, register-based variables were reg_COUNTER, and global data variables were (data_). While this scheme isn’t changing, we’re being much more intelligent about situations where additional information is available. > > > > For example, if a variable is passed to a function and a variable name is available, we can now make a much better guess for the variable name. This is most obvious in binaries with type libraries. > - > This isn’t the only style of default names. Binary Ninja also will name loop counters with simpler names like `i`, or `j`, `k`, etc (in the case of nested loops) > - https://github.com/Vector35/binaryninja-api/issues/2558 > > _Originally posted by @0xdevalias in https://github.com/pionxzh/wakaru/issues/34#issuecomment-1822263687_ --- > Was looking closer at the `sourcemap` spec today, and the `names` field jumped out at me as potentially useful: > > - https://tc39.es/source-map-spec/#names > - > `names`: a list of symbol names used by the `mappings` entry > - https://tc39.es/source-map-spec/#mappings > - > `mappings`: a string with the encoded mapping data (see _4.1 Mappings Structure_) > - https://tc39.es/source-map-spec/#mappings-structure > - > The `mappings` data is broken down as follows: > > > > - each group representing a line in the generated file is separated by a semicolon (`;`) > > - each segment is separated by a comma (`,`) > > - each segment is made up of `1`, `4`, or `5` variable length fields. > - It then goes on to describe the segment's in greater detail, but the specific part I was thinking could be relevant here would be this: > - > If present, the zero-based index into the `names` list associated with this segment. This field is a base 64 VLQ relative to the previous occurrence of this field unless this is the first occurrence of this field, in which case the whole value is represented. > > Obviously if there is a full `sourcemap` for the webapp, then `wakaru` isn't really needed anyway.. but what I was thinking of here is that in combination with module detection (see - https://github.com/pionxzh/wakaru/issues/41), if there are `sourcemaps`s available for that original module, then we could potentially extract the original function/variable/etc names from the `names` field of the `sourcemap`, and use them in a sort of 'smart-rename with sourcemap' type way. > > --- > > Another `sourcemap` related idea I had (which probably deserves it's own issue) is that it would be cool to be able to 'retroactively generate a `sourcemap`) for a webapp, based on the unminified output from `wakaru`; such that we could than take that `sourcemap`, and apply it to the original minified web app source for debugging the live app. > > **Edit:** Created a new issue to track this: > > - https://github.com/pionxzh/wakaru/issues/66 > > _Originally posted by @0xdevalias in https://github.com/pionxzh/wakaru/issues/34#issuecomment-1837858619_ --- > > It isn't very meaningful to support such a feature when you can access all the source code. > > @pionxzh I was specifically talking about it in terms of bundled modules (eg. React, etc), and not the unique web app code of the app itself. > > _Originally posted by @0xdevalias in https://github.com/pionxzh/wakaru/issues/34#issuecomment-1839791102_ --- > > You mean like, for popular open-source projects, we can put some sourcemap in our project / read from the chunk, and then reverse map the minified variable and function name back to normal? > > @pionxzh Similar to that, but probably not "put the sourcemap in our project" directly; but more process the sourcemaps from popular open-source projects and extract those details to an 'intermediary form'. That 'intermediary form' would be similar to the 'module map' file, as I described earlier in this thread: > > > When I was exploring PoC ideas for my own project previously, I was looking to generate a file similar to the 'module map' that this project is using; but instead of just for the names of modules, I wanted to be able to use it to provide a 'variable name map'. Though because the specific variables used in webpack/etc can change between builds, my thought was that first 'normalising' them to a 'known format' based on their context would make sense to do first. > > > > That could then be later enhanced/expanded by being able to pre-process these 'variable name mappings' for various open source projects in a way that could then be applied 'automagically' without the end user needing to first create them. > > > > It could also be enhanced by similar techniques such as what the `humanify` project does, by using LLMs/similar to generate suggested variable name mappings based on the code. > > > > _Originally posted by @0xdevalias in https://github.com/pionxzh/wakaru/issues/34#issuecomment-1807393509_ > > --- > > > A configuration table/profile can be provided to allow users to manually write correspondences. wakaru can simply include the rules of the better known packages. > > @StringKe _nods_, sounds like we are thinking about similar things here :) > > --- > > > Can you specify the content that you would expect to have? and the corresponding behavior > > @pionxzh For me personally, I haven't deeply thought through all the use cases in depth, but at a high level I basically want to be able to take a web app that is going to be re-built multiple times, and be able to have a 'config file' similar to the 'module mapping' that wakaru has/had; but that also allows me to specify the variable/function names ('symbols') that are used within it. > > The slightly more challenging part is that because the app will be re-built multiple times, the minified variables will change (sometimes every build), so we can't easily use those as the 'key' of the mapping. One idea I had for solving that is potentially by first renaming all of the variables based on a 'stable naming pattern' (eg. `func_*`, `arg_*`, `const_*`, etc; and then could just use a counter/similar based on the 'scope' it's being defined in) that would be generated based on the scope/type of the 'symbol', and would therefore be resilient to the minified variable names changing each build. Those 'stable intermediary names' could then potentially be used for the keys in the variable mapping. > > Though then we also need to figure out what level of 'granularity' makes sense to generate those 'stable intermediary names' at; as having a 1:1 mapping of those 'stable name scopes' to JS scopes could potentially end up being really noisy in the mapping file. So maybe using a 'higher abstracted scope' would make more sense (eg. at the module level or similar) > > My original hacky implementation of this in my own PoC code was using JS objects/JSON to map an explicit minified variable name to it's 'proper' name; but that broke because the minified names changed between builds. Even by implementing the 'stable naming pattern', if those 'stable names' included a 'counter' in them (eg. `func_1`, `const_8`, etc) we still probably wouldn't want to use those stable names directly as the key of an object, as if a new variable was added 'in between' in a later build, that would flow on to 'shifting' the 'counter' for every variable of a matching type afterwards, which would be a lot of effort to manually update in a mapping file. While I haven't thought too deeply about it, I think that by using an array in the mapping file, it should simplify things so that we only need to make a small change to 'fix the mappings' when a new variable is added that 'shifts' everything. > > Even by using the array concept in the mappings file, there is still some manual pain/effort involved in trying to keep the mapping 'up to date' in newer builds. That's what lead me into some of the deeper/more esoteric ideas/thinking around 'fingerprinting' that I expand on below. > > -- > > Another area I started looking into (but haven't deeply explored yet) for both figuring out how to map variable names to sections of code in a 'smart' way, and potentially also for module identification (see #41); is in the space of 'structural AST fingerprinting' or 'code similarity' algorithms and similar. (I realise that this is a rather deep/esoteric angle to be looking at this from, and that there are likely going to be far simpler/easier ways to implement the variable mapping/module identification in a 'good enough' way without going to this level of depth; but I'm curious to explore it regardless, to see if any good ideas come out of it) > > I haven't gotten too far in my reading yet (got distracted on other things), but the high level of my idea was that maybe we could generate an 'AST fingerprint' that isn't impacted by the variable/function/etc names ('symbols') changing during minification; and then use that as the basis for the 'key' in the 'mappings file'; as that fingerprint could theoretically still identify a 'scope' (which might be a literal JS scope, or might be a higher level abstraction that we decide makes sense; the most abstract being probably at the bundled module level) even if the bundler decides to move some functions around to a different module/etc. Then obviously if we were able to generate those 'resilient fingerprints' to identify code even when it's been minified, that would make perfect sense to apply to module detection/etc (see #41) as well. > > Some of the high level ideas / search terms that I was using to start my research in that area was things like: > > - AST fingerprinting > - Source code similarity fingerprinting > - Control flow graphs > - Call flow graphs > - Program dependence graph > - etc > > Here is a link dump of a bunch of the tabs I have open but haven't got around to reviewing in depth yet, RE: 'AST fingerprinting' / Code Similarity / etc: > > <details> > <summary>Unsorted/Unreviewed Initial Link Dump RE: 'AST fingerprinting' / Code Similarity</summary> > > - https://openreview.net/forum?id=BJxWx0NYPr > - > On the Effectiveness of Program Analysis in AI and Vice Versa > - https://dl.acm.org/doi/10.1145/3486860 > - > A Unified Framework for Code Representation Learning > - https://inria.hal.science/hal-01648996/document > - > A Survey of Program Slicing Techniques > - https://hal.science/hal-00627811/document > - > From Program Slicing to Abstract Interpretation > - https://igm.univ-mlv.fr/~chilowi/research/syntax_tree_fingerprinting/syntax_tree_fingerprinting_ICPC09.pdf > - > Syntax Tree Fingerprinting for Source Code Similarity Detection > - https://ieeexplore.ieee.org/document/9960266 > - > Exploring the Landscape of Deep Learning for Source Code Analysis > - https://en.wikipedia.org/wiki/Program_dependence_graph > - > Program Dependence Graph - Wikipedia > - https://digitalcommons.calpoly.edu/cgi/viewcontent.cgi?article=3437&context=theses > - > Enhancing Software Reliability Using Program Analysis Techniques > - https://dl.acm.org/doi/pdf/10.1145/1286821.1286826 > - > Static Analysis of Executables to Detect Malicious Patterns > - https://patents.google.com/patent/US9459861B1/en > - > US Patent US9459861B1: Techniques for Code Analysis and Optimization > - https://www.unomaha.edu/college-of-information-science-and-technology/research-labs/_files/software-nsf.pdf > - > Software Engineering Research and Practice: NSF Workshop Report > - https://www.computer.org/csdl/journal/ts/2023/08/10125077/1Nc4Vd4vb7W > - > Advances in Automated Program Repair: IEEE Transactions > - https://stackoverflow.com/questions/7283702/assembly-level-function-fingerprint > - > Stack Overflow: Assembly-Level Function Fingerprinting > - https://en.wikipedia.org/wiki/Control-flow_graph > - > Control-Flow Graph - Wikipedia > - https://www.kapravelos.com/publications/fptechniques-www23.pdf > - > Novel Techniques for Malware and Vulnerability Analysis in Web Applications > - https://pubmed.ncbi.nlm.nih.gov/12197666/ > - > PubMed: Program Analysis Techniques for Software Debugging > - https://stackoverflow.com/questions/15087195/data-flow-graph-construction > - > Stack Overflow: Data Flow Graph Construction in Programming > - https://www.cs.columbia.edu/~suman/secure_sw_devel/Basic_Program_Analysis_CF.pdf > - > Basic Program Analysis Techniques for Secure Software Development > - https://codeql.github.com/docs/writing-codeql-queries/about-data-flow-analysis/ > - > CodeQL Guide: Understanding Data Flow Analysis for Secure Software Development > - https://codereview.stackexchange.com/questions/276387/call-flow-graph-from-python-abstract-syntax-tree > - > Code Review Stack Exchange: Call Flow Graph from Python AST > - https://clang.llvm.org/docs/DataFlowAnalysisIntro.html > - > LLVM Clang Documentation: Introduction to Data Flow Analysis > - https://www.researchgate.net/figure/AST-based-program-graph-with-additional-control-and-data-flow-edges_fig2_370980383 > - > ResearchGate: AST-Based Program Graphs with Control and Data Flow Edges > > </details> > > -- > > Another idea I've had, but only lightly explored so far, is looking into how various projects like Terser, Webpack, etc choose their minified variable names in general; but also how they handle 'stable minified variables' between builds (which is something that I know at least Webpack has some concept of). My thought there is that by understanding how they implement their own 'stable minified variables between builds', that we might be able to leverage to either a) do similar, or b) be able to reverse engineer that in a way that might be able to be 'retroactively applied' on top of an existing minified project that didn't use 'stable minified variables', to 'stabilise' them. > > _Originally posted by @0xdevalias in https://github.com/pionxzh/wakaru/issues/34#issuecomment-1843850057_ -
0xdevalias revised this gist
Dec 6, 2023 . 1 changed file with 5 additions and 0 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -38,6 +38,11 @@ <!-- TOC end --> **Other files in this gist:** - [`fingerprinting-minified-javascript-libraries.md`](#file-fingerprinting-minified-javascript-libraries-md) - > Fingerprinting Minified JavaScript Libraries ## Tools ### Unsorted -
0xdevalias revised this gist
Dec 4, 2023 . 1 changed file with 183 additions and 175 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -9,7 +9,9 @@ - [wakaru](#wakaru) - [webcrack](#webcrack) - [ast-grep](#ast-grep) - [Restringer](#restringer) - [debundle + related](#debundle--related) - [joern](#joern) - [Blogs / Articles / etc](#blogs--articles--etc) - [Libraries / Helpers](#libraries--helpers) - [Unsorted](#unsorted-1) @@ -93,24 +95,6 @@ - > Reverse many of the transformations applied by minifiers and naïve obfuscators - https://github.com/shapesecurity/unminify/#safety-levels - https://unminify.io/ - https://github.com/lelinhtinh/de4js - > JavaScript Deobfuscator and Unpacker - https://lelinhtinh.github.io/de4js/ @@ -136,96 +120,6 @@ - > The Cyber Swiss Army Knife - a web app for encryption, encoding, compression and data analysis - https://gchq.github.io/CyberChef/ - Javascrpt Parser: https://gchq.github.io/CyberChef/#recipe=JavaScript_Parser(false,false,false,false,false) - https://github.com/dandavison/delta - > A syntax-highlighting pager for git, diff, and grep output - > (the package is called "git-delta" in most package managers, but the executable is just delta) @@ -243,70 +137,9 @@ - > When comparing two JavaScript files, standard diff tools compare the two files line-by-line and output the lines on which the files differ. This tool does not compare the characters of the source files directly but their abstract representation - their abstract syntax trees. - > This enables you to have more meaningfull diffs between files which may be very simmilar but have different source code formatting. - > When patching, astii patch will regenerate (`original --> AST --> generate`) the source file and patch it with the provided diff. - https://www.blueclosure.com/product/bc-detect - > BC Detect (DOMinatorPro NG) helps security testers to analyse and automatically discover DOM Based Cross Site Scripting issues thanks to its Hybrid IAST Engine together with the Smart Fuzzer module. - https://www.blueclosure.com/page/features ### wakaru @@ -474,6 +307,27 @@ > * ruff: [astral-sh/ruff@`main`/crates/ruff_python_semantic/src](https://github.com/astral-sh/ruff/tree/main/crates/ruff_python_semantic/src?rgh-link-date=2023-06-09T22%3A04%3A40Z) - https://www.reddit.com/r/rust/comments/13eg738/meet_astgrep_a_rustbased_tool_for_code_searching/ ### Restringer - https://github.com/PerimeterX/restringer - > Restringer - > Deobfuscate Javascript and reconstruct strings. Simplify cumbersome logic where possible while adhering to scope limitations. - https://restringer.tech/ (Playground) - https://github.com/PerimeterX/restringer/tree/main/src/processors - > Processors > Processors are a collection of methods meant to prepare the script for obfuscation, removing anti-debugging traps and performing any required modifications before (preprocessors) or after (postprocessors) the main deobfuscation process. > > The processors are created when necessary and are lazily loaded when a specific obfuscation type was detected which requires these additional processes. - https://github.com/PerimeterX/restringer/tree/main/src/modules/safe - https://github.com/PerimeterX/restringer/tree/main/src/modules/unsafe - https://github.com/PerimeterX/obfuscation-detector - > Obfuscation Detector - > Detect different types of JS obfuscation by their AST structure. - https://github.com/PerimeterX/flast - > flAST - FLat Abstract Syntax Tree > Flatten an Abstract Syntax Tree by placing all the nodes in a single flat array. - > Provides a flat Abstract Syntax Tree and an Arborist to trim and modify the tree ### debundle + related - https://github.com/1egoman/debundle @@ -514,6 +368,36 @@ > > Debundles a large React Native bundle by walking the compiled AST and extracts individual module declarations and writes them to their own modules & attempts to resolve dependeny import relationships. ### joern - https://joern.io/ - > The Bug Hunter's Workbench - > Query: Uncover attack surface, sloppy coding practices, and variants of known vulnerabilities using an interactive code analysis shell. Joern supports C, C++, LLVM bitcode, x86 binaries (via Ghidra), JVM bytecode (via Soot), and Javascript. Python, Java source code, Kotlin, and PHP support coming soon. - > Automate: Wrap your queries into custom code scanners and share them with the community or run existing Joern-based scanners in your CI. - > Integrate: Use Joern as a library to power your own code analysis tools or as a component via the REST API. - https://github.com/joernio/joern - > Open-source code analysis platform for C/C++/Java/Binary/Javascript/Python/Kotlin based on code property graphs. - > Joern is a platform for analyzing source code, bytecode, and binary executables. It generates code property graphs (CPGs), a graph representation of code for cross-language code analysis. Code property graphs are stored in a custom graph database. This allows code to be mined using search queries formulated in a Scala-based domain-specific query language. Joern is developed with the goal of providing a useful tool for vulnerability discovery and research in static program analysis. - https://docs.joern.io/ - > Joern is a platform for robust analysis of source code, bytecode, and binary code. It generates code property graphs, a graph representation of code for cross-language code analysis. Code property graphs are stored in a custom graph database. This allows code to be mined using search queries formulated in a Scala-based domain-specific query language. Joern is developed with the goal of providing a useful tool for vulnerability discovery and research in static program analysis. - > The core features of Joern are: > - Robust parsing. Joern allows importing code even if a working build environment cannot be supplied or parts of the code are missing. > - Code Property Graphs. Joern creates semantic code property graphs from the fuzzy parser output and stores them in an in-memory graph database. SCPGs are a language-agnostic intermediate representation of code designed for query-based code analysis. > - Taint Analysis. Joern provides a taint-analysis engine that allows the propagation of attacker-controlled data in the code to be analyzed statically. > - Search Queries. Joern offers a strongly-typed Scala-based extensible query language for code analysis based on Gremlin-Scala. This language can be used to manually formulate search queries for vulnerabilities as well as automatically infer them using machine learning techniques. > - Extendable via CPG passes. Code property graphs are multi-layered, offering information about code on different levels of abstraction. Joern comes with many default passes, but also allows users to add passes to include additional information in the graph, and extend the query language accordingly. - https://docs.joern.io/code-property-graph/ - https://docs.joern.io/cpgql/data-flow-steps/ - https://docs.joern.io/export/ - > Joern can create the following graph representations for C/C++ code: > - Abstract Syntax Trees (AST) > - Control Flow Graphs (CFG) > - Control Dependence Graphs (CDG) > - Data Dependence Graphs (DDG) > - Program Dependence graphs (PDG) > - Code Property Graphs (CPG14) > - Entire graph, i.e. convert to a different graph format (ALL) ## Blogs / Articles / etc - https://thejunkland.com/blog/using-llms-to-reverse-javascript-minification.html @@ -743,9 +627,39 @@ - > CLI to generate an interactive graph of functions and calls from your TypeScript files - https://github.com/Deskbot/TS-Call-Graph - > A program that generates a graph of the methods and attributes of a TypeScript class - https://github.com/julianjensen/ast-flow-graph - > ast-flow-graph - > Creates a CFG from JavaScript source code. - > This module will read one or more JavaScript source files and produce CFGs (Control Flow Graphs) of the code. - Uses espree, escope, estraverse, etc - https://github.com/isaacs/yallist - > Yet Another Linked List > > There are many doubly-linked list implementations like it, but this one is mine. > > For when an array would be too big, and a Map can't be iterated in reverse order. - https://github.com/julianjensen/traversals - > Small module for graph traversals, supporting DFS and BFS with niceties added for pre- and post-order, including their reverses. - Some notes from ChatGPT: - > Provides a small module designed for performing graph traversal operations, specifically Depth-First Search (DFS) and Breadth-First Search (BFS). It includes additional features such as pre-order and post-order traversals, as well as their reverse versions, to enhance the functionality of these standard graph traversal techniques. - https://github.com/julianjensen/dominators - > Various dominator tree algorithms - > It implements two different methods for finding the immediate dominators of a graph. - Some notes from ChatGPT: - > A dominator tree is a concept used in computer science, particularly in the field of compiler design and program analysis. To understand a dominator tree, let's first look at the concept of dominators in a control flow graph (CFG). > > In a CFG, which represents the flow of control in a program, a node \( A \) is said to dominate another node \( B \) if every path from the start node of the graph to \( B \) must go through \( A \). In other words, \( A \) dominates \( B \) if \( A \) is always encountered before \( B \) when traversing the graph from the start node. > > The concept becomes more nuanced with the idea of immediate dominators. An immediate dominator of a node \( B \) is the last dominator on any path from the start node to \( B \). > > Now, a dominator tree is a tree structure that represents these dominance relationships within a CFG. In this tree: > > - Each node corresponds to a node in the original CFG. > - There is a directed edge from node \( A \) to node \( B \) if \( A \) is the immediate dominator of \( B \) in the CFG. > > A dominator tree generator, therefore, is a tool or an algorithm that constructs the dominator tree from a given control flow graph. This tool is essential in optimizing compilers and in various program analysis tasks, where understanding the dominance relationships helps in transformations like loop optimization, dead code elimination, and more sophisticated analyses like static single assignment (SSA) form conversion. > > This concept is closely related to computer science and software engineering, particularly in areas concerning compiler construction and code optimization. Given your background in software engineering and ethical hacking, this knowledge could be particularly useful in understanding code structure and flow, especially when analyzing or optimizing complex software systems. - https://github.com/dochne/wappalyzer - > Wappalyzer identifies technologies on websites, such as CMS, web frameworks, ecommerce platforms, JavaScript libraries, analytics tools and more. - > The last commit of Wappalyzer before it went private @@ -811,6 +725,100 @@ - > Turn your ES5 code into readable ES6. Lebab does the opposite of what Babel does. - > Lebab transpiles your ES5 code to ES6/ES7. It does exactly the opposite of what Babel does. - https://lebab.github.io/ (Playground) - https://github.com/EricSmekens/jsep - > `jsep`: A Tiny JavaScript Expression Parser > `jsep` is a simple expression parser written in JavaScript. It can parse JavaScript expressions but not operations. The difference between expressions and operations is akin to the difference between a cell in an Excel spreadsheet vs. a proper JavaScript program. - https://ericsmekens.github.io/jsep/ - https://github.com/oxc-project/oxc - > Oxc > The Oxidation Compiler is creating a suite of high-performance tools for JavaScript and TypeScript. > > Oxc is building a parser, linter, formatter, transpiler, minifier, resolver ... all written in Rust. - https://github.com/oxc-project/oxc#-ast-and-parser - > AST and Parser > Oxc maintains its own AST and parser, which is by far the fastest and most conformant JavaScript and TypeScript (including JSX and TSX) parser written in Rust. - > While many existing JavaScript tools rely on [estree](https://github.com/estree/estree) as their AST specification, a notable drawback is its abundance of ambiguous nodes. This ambiguity often leads to confusion during development with [estree](https://github.com/estree/estree). > > The Oxc AST differs slightly from the [estree](https://github.com/estree/estree) AST by removing ambiguous nodes and introducing distinct types. For example, instead of using a generic [estree](https://github.com/estree/estree) `Identifier`, the Oxc AST provides specific types such as `BindingIdentifier`, `IdentifierReference`, and `IdentifierName`. This clear distinction greatly enhances the development experience by aligning more closely with the ECMAScript specification. - https://oxc-project.github.io/ - https://oxc-project.github.io/oxc/playground/ - https://oxc-project.github.io/docs/guide/usage/parser.html - > Parser > - 2x faster then [SWC](https://swc.rs/) parser > - By far the fastest and most conformant JavaScript and TypeScript (including JSX and TSX) parser written in Rust - https://github.com/oxc-project/bench-javascript-parser-written-in-rust - > Benchmark for Oxc, Swc and Rome parser - > The umbrella crate [`oxc`](https://docs.rs/oxc) exports all public crates from this repository - > The AST and parser crates [`oxc_ast`](https://docs.rs/oxc_ast) and [`oxc_parser`](https://docs.rs/oxc_parser) are production ready - https://docs.rs/oxc_parser/latest/oxc_parser/#conformance - > The parser parses all of Test262 and most of Babel and TypeScript parser conformance tests. - https://github.com/oxc-project/oxc/tree/main/tasks/coverage - https://oxc-project.github.io/javascript-parser-in-rust/ - > Write a JavaScript Parser in Rust > > Rust, JavaScript, and parsers are all hard to learn, let's combine these three and challenge ourselves to write a JavaScript parser in Rust. > > This will be the guide for you if you are interested in learning Rust, parsers, or would like to contribute to [oxc](https://github.com/boshen/oxc), [swc](https://swc.rs/) or [Biome](https://biomejs.dev/) in the near future. > > The guide will cover all the basic topics of writing a JavaScript parser in rust. The tutorials will explain some topics in more depth. - https://oxc-project.github.io/javascript-parser-in-rust/blog/ - https://oxc-project.github.io/javascript-parser-in-rust/blog/rome/ - > Rome Tools > Rome uses a different set of techniques for parsing JavaScript and TypeScript. This tutorial summarizes them in learning order for better understanding. - > The Rome codebase was rewritten from TypeScript to Rust, see Rome will be rewritten in Rust > The decision was made after talking to the author of `rslint` and `rust-analyzer` > `rust-analyzer` proved that IDE-centric tools built around concrete syntax tree are possible > `rslint` proved that it is possible to write a JavaScript parser in Rust, with the same base libraries as rust-analyzer - > The base library is called [rowan](https://github.com/rust-analyzer/rowan), see [overview of rowan](https://github.com/rust-lang/rust-analyzer/blob/master/docs/dev/syntax.md) > Rowan, also known as red-green trees, is named after the real green [rowan tree](https://en.wikipedia.org/wiki/Rowan) that makes red berries > The origin of red-green trees is described in this [blog post](https://ericlippert.com/2012/06/08/red-green-trees/), by the authors of the C# programming language > The whole point of rowan is to define a lossless concrete syntax tree (CST) that describes all the details of the source code and provides a set of traversal APIs (parent, children, siblings, etc) > Read the advantage of having a CST over an AST: [Pure AST based linting sucks](https://rdambrosio016.github.io/rust/2020/09/18/pure-ast-based-linting-sucks.html) > CST provides the ability to build a fully recoverable parser - https://craftinginterpreters.com/ - > Ever wanted to make your own programming language or wondered how they are designed and built? If so, this book is for you. - > Crafting Interpreters contains everything you need to implement a full-featured, efficient scripting language. You’ll learn both high-level concepts around parsing and semantics and gritty details like bytecode representation and garbage collection. Your brain will light up with new ideas, and your hands will get dirty and calloused. It’s a blast. > > Starting from `main()`, you build a language that features rich syntax, dynamic typing, garbage collection, lexical scope, first-class functions, closures, classes, and inheritance. All packed into a few thousand lines of clean, fast code that you thoroughly understand because you write each one yourself. - https://github.com/kaleidawave/ezno - > A JavaScript compiler and TypeScript checker written in Rust with a focus on static analysis and runtime performance - https://kaleidawave.github.io/posts/introducing-ezno/ - > Introducing Ezno - > Ezno is an experimental compiler I have been working on and off for a while. In short, it is a JavaScript compiler featuring checking, correctness and performance for building full-stack (rendering on the client and server) websites. - https://kaleidawave.github.io/posts/ezno-23/ - > Ezno in '23 - https://github.com/boa-dev/boa - > Boa is an embeddable and experimental Javascript engine written in Rust. Currently, it has support for some of the language. - > This is an experimental Javascript lexer, parser and interpreter written in Rust. Currently, it has support for some of the language. - https://boajs.dev/ - > Boa is an experimental Javascript lexer, parser and compiler written in Rust. Currently, it has support for some of the language. It can be embedded in Rust projects fairly easily and also used from the command line. Boa also exists to serve as a Rust implementation of the EcmaScript specification, there will be areas where we can utilise Rust and its fantastic ecosystem to make a fast, concurrent and safe engine. - https://boajs.dev/boa/playground/ - https://boajs.dev/posts/2022-10-24-boa-usage/ - > Adding a JavaScript interpreter to your Rust project - https://boajs.dev/boa/dev/bench/ - > Boa Benchmarks - https://github.com/boa-dev/boa/blob/main/boa_engine/benches/full.rs#L9 - https://github.com/bheisler/criterion.rs - > Statistics-driven benchmarking library for Rust - > Criterion.rs helps you write fast code by detecting and measuring performance improvements or regressions, even small ones, quickly and accurately. You can optimize with confidence, knowing how each change affects the performance of your code. - https://docs.rs/criterion/latest/criterion/ - https://boajs.dev/boa/test262/ - > EcmaScript conformance test results for Boa - https://github.com/rust-analyzer/rowan - > Rowan is a library for lossless syntax trees, inspired in part by Swift's libsyntax. - https://github.com/mozilla-spidermonkey/jsparagus - > Experimental JS parser-generator project. - > jsparagus - A JavaScript parser written in Rust > jsparagus is intended to replace the JavaScript parser in Firefox. - https://github.com/mozilla-spidermonkey/jsparagus#benchmarking - > Benchmarking - https://github.com/mozilla-spidermonkey/jsparagus#limitations - > Limitations - https://github.com/ratel-rust/ratel-core - > High performance JavaScript to JavaScript compiler with a Rust core - https://maciej.codes/ratel-wasm/ (REPL / Playground) - https://github.com/ratel-rust/ratel-cli - > A command-line interface for Ratel, a high performance JavaScript to JavaScript compiler with a Rust core ### Recast + related -
0xdevalias revised this gist
Dec 4, 2023 . 1 changed file with 74 additions and 70 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -8,6 +8,7 @@ - [Unsorted](#unsorted) - [wakaru](#wakaru) - [webcrack](#webcrack) - [ast-grep](#ast-grep) - [debundle + related](#debundle--related) - [Blogs / Articles / etc](#blogs--articles--etc) - [Libraries / Helpers](#libraries--helpers) @@ -135,76 +136,6 @@ - > The Cyber Swiss Army Knife - a web app for encryption, encoding, compression and data analysis - https://gchq.github.io/CyberChef/ - Javascrpt Parser: https://gchq.github.io/CyberChef/#recipe=JavaScript_Parser(false,false,false,false,false) - https://github.com/oxc-project/oxc - > Oxc > The Oxidation Compiler is creating a suite of high-performance tools for JavaScript and TypeScript. @@ -470,6 +401,79 @@ - https://github.com/e9x/krunker-decompiler/blob/master/src/libDecompile.ts - https://github.com/e9x/krunker-decompiler/blob/master/src/libRenameVars.ts ### ast-grep - https://github.com/ast-grep/ast-grep - > A CLI tool for code structural search, lint and rewriting. Written in Rust - > ast-grep is a AST-based tool to search code by pattern code. Think it as your old-friend grep but it matches AST nodes instead of text. You can write patterns as if you are writing ordinary code. It will match all code that has the same syntactical structure. You can use `$` sign + upper case letters as wildcard, e.g. `$MATCH`, to match any single AST node. Think it as REGEX dot `.`, except it is not textual. - https://github.com/ast-grep/ast-grep/tree/main/npm - > @ast-grep/cli > `ast-grep(sg)` is a CLI tool for code structural search, lint, and rewriting. - https://github.com/ast-grep/ast-grep/tree/main/crates - https://github.com/ast-grep/ast-grep/blob/main/benches - https://github.com/ast-grep/ast-grep/blob/main/benches/bench.ts#L37-L101 - https://ast-grep.github.io/ - https://ast-grep.github.io/playground.html - https://ast-grep.github.io/reference/playground.html - > ast-grep Playground Manual > The ast-grep playground is an online tool that allows you to try out ast-grep without installing anything on your machine. You can write code patterns and see how they match your code in real time. You can also apply rewrite rules to modify your code based on the patterns. > > The playground is a great way to learn ast-grep, debug patterns/rules, report bugs and showcase ast-grep's capabilities. - https://ast-grep.github.io/guide/introduction.html - https://ast-grep.github.io/reference/languages.html - > List of Languages with Built-in Support - https://ast-grep.github.io/reference/cli.html - https://ast-grep.github.io/reference/api.html - > API Reference > ast-grep currently has an experimental API for Node.js - https://ast-grep.github.io/guide/tooling-overview.html#editor-integration - https://marketplace.visualstudio.com/items?itemName=ast-grep.ast-grep-vscode&ssr=false#overview - > ast-grep-vscode > VSCode extension package for ast-grep language server - https://github.com/ast-grep/ast-grep/issues/80 - > Using ast-grep as a library - https://ast-grep.github.io/guide/api-usage/py-api.html - > Python API > `ast-grep`'s Python API is powered by PyO3. You can write Python to programmatically inspect and change syntax trees. - https://github.com/ast-grep/ast-grep/issues/524 - > More powerful `tree-sitter` analysis - https://github.com/microsoft/vscode-anycode - > Anycode > A Tree-sitter-based language extension that inaccurately implements popular features like "Outline & Breadcrumbs", "Go to Symbol in Workspace", "Document Highlights" and more. This extension should be used when running in enviroments that don't allow for running actual language services, like https://github.dev or https://vscode.dev. - https://code.visualstudio.com/docs/editor/editingevolved#_go-to-symbol - https://microsoft.github.io/monaco-editor/typedoc/interfaces/languages.DocumentSymbolProvider.html - https://github.com/ast-grep/ast-grep/issues/334 - > [pipedream] Add control flow / data dependency operators to ast-grep - > inspired by docs.joern.io/code-property-graph, or more specifically, docs.joern.io/cpgql/data-flow-steps - > State of Art: > > * [arxiv.org/pdf/2208.07461.pdf](https://arxiv.org/pdf/2208.07461.pdf) > > * parser: tree-sitter > * cfg/data flow: built for Python > * [nickgregory.me/post/2022/07/02/go-code-as-a-graph](https://nickgregory.me/post/2022/07/02/go-code-as-a-graph/) > > * [kallsyms/go-graph](https://github.com/kallsyms/go-graph) > * parser: go > * cfg/dataflow: golangx/tool > * golang team's tooling is crazy... > * [docs.rs/tree-sitter-graph/0.10.4/tree_sitter_graph/reference](https://docs.rs/tree-sitter-graph/0.10.4/tree_sitter_graph/reference/) > > * parser: tree-sitter > * cfg/data flow: N/A. It only supports graph construction. > * joern > > * parser: graal based. e.g. graalJS or custom parser generator > * e.g. Python [joernio/joern@`7e66155`/joern-cli/frontends/pysrc2cpg/pythonGrammar.jj#L10](https://github.com/joernio/joern/blob/7e6615548fa06e94c3307da2a85709295d9f660f/joern-cli/frontends/pysrc2cpg/pythonGrammar.jj#L10) > * cfg: based on a generic-ast. [joernio/joern@`7e66155`/joern-cli/frontends/x2cpg/src/main/scala/io/joern/x2cpg](https://github.com/joernio/joern/tree/7e6615548fa06e94c3307da2a85709295d9f660f/joern-cli/frontends/x2cpg/src/main/scala/io/joern/x2cpg) > > Not related but worth looking > > * eslint: [eslint/eslint@`main`/lib/linter/code-path-analysis/code-path-analyzer.js](https://github.com/eslint/eslint/blob/main/lib/linter/code-path-analysis/code-path-analyzer.js?rgh-link-date=2023-06-09T22%3A04%3A40Z) > * oxc: [Boshen/oxc@`main`/crates/oxc_semantic2/src/reference.rs](https://github.com/Boshen/oxc/blob/main/crates/oxc_semantic2/src/reference.rs?rgh-link-date=2023-06-09T22%3A04%3A40Z) > * ruff: [astral-sh/ruff@`main`/crates/ruff_python_semantic/src](https://github.com/astral-sh/ruff/tree/main/crates/ruff_python_semantic/src?rgh-link-date=2023-06-09T22%3A04%3A40Z) - https://www.reddit.com/r/rust/comments/13eg738/meet_astgrep_a_rustbased_tool_for_code_searching/ ### debundle + related - https://github.com/1egoman/debundle -
0xdevalias revised this gist
Dec 4, 2023 . 1 changed file with 4 additions and 0 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -372,6 +372,10 @@ > A dominator tree generator, therefore, is a tool or an algorithm that constructs the dominator tree from a given control flow graph. This tool is essential in optimizing compilers and in various program analysis tasks, where understanding the dominance relationships helps in transformations like loop optimization, dead code elimination, and more sophisticated analyses like static single assignment (SSA) form conversion. > > This concept is closely related to computer science and software engineering, particularly in areas concerning compiler construction and code optimization. Given your background in software engineering and ethical hacking, this knowledge could be particularly useful in understanding code structure and flow, especially when analyzing or optimizing complex software systems. - https://github.com/EricSmekens/jsep - > `jsep`: A Tiny JavaScript Expression Parser > `jsep` is a simple expression parser written in JavaScript. It can parse JavaScript expressions but not operations. The difference between expressions and operations is akin to the difference between a cell in an Excel spreadsheet vs. a proper JavaScript program. - https://ericsmekens.github.io/jsep/ ### wakaru -
0xdevalias revised this gist
Dec 1, 2023 . 1 changed file with 2 additions and 0 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -850,6 +850,8 @@ - > What can be said about recast can probably also be said to a lesser degree about `ast-types` - https://github.com/codemod-js/codemod - > codemod rewrites JavaScript and TypeScript using babel plugins - https://github.com/unjs/magicast - > Programmatically modify JavaScript and TypeScript source codes with a simplified, elegant and familiar syntax powered by recast and babel. ### estools + related -
0xdevalias revised this gist
Dec 1, 2023 . 1 changed file with 6 additions and 0 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -138,6 +138,12 @@ - https://github.com/ast-grep/ast-grep - > A CLI tool for code structural search, lint and rewriting. Written in Rust - > ast-grep is a AST-based tool to search code by pattern code. Think it as your old-friend grep but it matches AST nodes instead of text. You can write patterns as if you are writing ordinary code. It will match all code that has the same syntactical structure. You can use `$` sign + upper case letters as wildcard, e.g. `$MATCH`, to match any single AST node. Think it as REGEX dot `.`, except it is not textual. - https://github.com/ast-grep/ast-grep/tree/main/npm - > @ast-grep/cli > `ast-grep(sg)` is a CLI tool for code structural search, lint, and rewriting. - https://github.com/ast-grep/ast-grep/tree/main/crates - https://github.com/ast-grep/ast-grep/blob/main/benches - https://github.com/ast-grep/ast-grep/blob/main/benches/bench.ts#L37-L101 - https://ast-grep.github.io/ - https://ast-grep.github.io/playground.html - https://ast-grep.github.io/reference/playground.html -
0xdevalias revised this gist
Dec 1, 2023 . 1 changed file with 18 additions and 0 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -1646,6 +1646,24 @@ In addition to the links directly below, also make sure to check out the various > The improvements to start up latency you can expect will depend on how much initialization work your WebAssembly module needs to do before it's ready. Some initial benchmarking shows between 1.35 to 6.00 times faster instantiation and initialization with Wizer, depending on the workload - https://wingolog.org/archives/2022/08/18/just-in-time-code-generation-within-webassembly - > just-in-time code generation within webassembly - https://github.com/WebAssembly/wabt - > The WebAssembly Binary Toolkit - > WABT (we pronounce it "wabbit") is a suite of tools for WebAssembly, including: > > - `wat2wasm`: translate from WebAssembly text format to the WebAssembly binary format > - `wasm2wat`: the inverse of wat2wasm, translate from the binary format back to the text format (also known as a .wat) > - `wasm-objdump`: print information about a wasm binary. Similiar to objdump. > - `wasm-interp`: decode and run a WebAssembly binary file using a stack-based interpreter > - `wasm-decompile`: decompile a wasm binary into readable C-like syntax. > - `wat-desugar`: parse .wat text form as supported by the spec interpreter (s-expressions, flat syntax, or mixed) and print "canonical" flat format > - `wasm2c`: convert a WebAssembly binary file to a C source and header > - `wasm-strip`: remove sections of a WebAssembly binary file > - `wasm-validate`: validate a file in the WebAssembly binary format > - `wast2json`: convert a file in the wasm spec test format to a JSON file and associated wasm binary files > - `wasm-stats`: output stats for a module > - `spectest-interp`: read a Spectest JSON file, and run its tests in the interpreter > > These tools are intended for use in (or for development of) toolchains or other systems that want to manipulate WebAssembly files. Unlike the WebAssembly spec interpreter (which is written to be as simple, declarative and "speccy" as possible), they are written in C/C++ and designed for easier integration into other systems. Unlike Binaryen these tools do not aim to provide an optimization platform or a higher-level compiler target; instead they aim for full fidelity and compliance with the spec (e.g. 1:1 round-trips with no changes to instructions). ## My ChatGPT Research / Conversations -
0xdevalias revised this gist
Dec 1, 2023 . 1 changed file with 90 additions and 0 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -199,6 +199,96 @@ > * oxc: [Boshen/oxc@`main`/crates/oxc_semantic2/src/reference.rs](https://github.com/Boshen/oxc/blob/main/crates/oxc_semantic2/src/reference.rs?rgh-link-date=2023-06-09T22%3A04%3A40Z) > * ruff: [astral-sh/ruff@`main`/crates/ruff_python_semantic/src](https://github.com/astral-sh/ruff/tree/main/crates/ruff_python_semantic/src?rgh-link-date=2023-06-09T22%3A04%3A40Z) - https://www.reddit.com/r/rust/comments/13eg738/meet_astgrep_a_rustbased_tool_for_code_searching/ - https://github.com/oxc-project/oxc - > Oxc > The Oxidation Compiler is creating a suite of high-performance tools for JavaScript and TypeScript. > > Oxc is building a parser, linter, formatter, transpiler, minifier, resolver ... all written in Rust. - https://github.com/oxc-project/oxc#-ast-and-parser - > AST and Parser > Oxc maintains its own AST and parser, which is by far the fastest and most conformant JavaScript and TypeScript (including JSX and TSX) parser written in Rust. - > While many existing JavaScript tools rely on [estree](https://github.com/estree/estree) as their AST specification, a notable drawback is its abundance of ambiguous nodes. This ambiguity often leads to confusion during development with [estree](https://github.com/estree/estree). > > The Oxc AST differs slightly from the [estree](https://github.com/estree/estree) AST by removing ambiguous nodes and introducing distinct types. For example, instead of using a generic [estree](https://github.com/estree/estree) `Identifier`, the Oxc AST provides specific types such as `BindingIdentifier`, `IdentifierReference`, and `IdentifierName`. This clear distinction greatly enhances the development experience by aligning more closely with the ECMAScript specification. - https://oxc-project.github.io/ - https://oxc-project.github.io/oxc/playground/ - https://oxc-project.github.io/docs/guide/usage/parser.html - > Parser > - 2x faster then [SWC](https://swc.rs/) parser > - By far the fastest and most conformant JavaScript and TypeScript (including JSX and TSX) parser written in Rust - https://github.com/oxc-project/bench-javascript-parser-written-in-rust - > Benchmark for Oxc, Swc and Rome parser - > The umbrella crate [`oxc`](https://docs.rs/oxc) exports all public crates from this repository - > The AST and parser crates [`oxc_ast`](https://docs.rs/oxc_ast) and [`oxc_parser`](https://docs.rs/oxc_parser) are production ready - https://docs.rs/oxc_parser/latest/oxc_parser/#conformance - > The parser parses all of Test262 and most of Babel and TypeScript parser conformance tests. - https://github.com/oxc-project/oxc/tree/main/tasks/coverage - https://oxc-project.github.io/javascript-parser-in-rust/ - > Write a JavaScript Parser in Rust > > Rust, JavaScript, and parsers are all hard to learn, let's combine these three and challenge ourselves to write a JavaScript parser in Rust. > > This will be the guide for you if you are interested in learning Rust, parsers, or would like to contribute to [oxc](https://github.com/boshen/oxc), [swc](https://swc.rs/) or [Biome](https://biomejs.dev/) in the near future. > > The guide will cover all the basic topics of writing a JavaScript parser in rust. The tutorials will explain some topics in more depth. - https://oxc-project.github.io/javascript-parser-in-rust/blog/ - https://oxc-project.github.io/javascript-parser-in-rust/blog/rome/ - > Rome Tools > Rome uses a different set of techniques for parsing JavaScript and TypeScript. This tutorial summarizes them in learning order for better understanding. - > The Rome codebase was rewritten from TypeScript to Rust, see Rome will be rewritten in Rust > The decision was made after talking to the author of `rslint` and `rust-analyzer` > `rust-analyzer` proved that IDE-centric tools built around concrete syntax tree are possible > `rslint` proved that it is possible to write a JavaScript parser in Rust, with the same base libraries as rust-analyzer - > The base library is called [rowan](https://github.com/rust-analyzer/rowan), see [overview of rowan](https://github.com/rust-lang/rust-analyzer/blob/master/docs/dev/syntax.md) > Rowan, also known as red-green trees, is named after the real green [rowan tree](https://en.wikipedia.org/wiki/Rowan) that makes red berries > The origin of red-green trees is described in this [blog post](https://ericlippert.com/2012/06/08/red-green-trees/), by the authors of the C# programming language > The whole point of rowan is to define a lossless concrete syntax tree (CST) that describes all the details of the source code and provides a set of traversal APIs (parent, children, siblings, etc) > Read the advantage of having a CST over an AST: [Pure AST based linting sucks](https://rdambrosio016.github.io/rust/2020/09/18/pure-ast-based-linting-sucks.html) > CST provides the ability to build a fully recoverable parser - https://craftinginterpreters.com/ - > Ever wanted to make your own programming language or wondered how they are designed and built? If so, this book is for you. - > Crafting Interpreters contains everything you need to implement a full-featured, efficient scripting language. You’ll learn both high-level concepts around parsing and semantics and gritty details like bytecode representation and garbage collection. Your brain will light up with new ideas, and your hands will get dirty and calloused. It’s a blast. > > Starting from `main()`, you build a language that features rich syntax, dynamic typing, garbage collection, lexical scope, first-class functions, closures, classes, and inheritance. All packed into a few thousand lines of clean, fast code that you thoroughly understand because you write each one yourself. - https://github.com/kaleidawave/ezno - > A JavaScript compiler and TypeScript checker written in Rust with a focus on static analysis and runtime performance - https://kaleidawave.github.io/posts/introducing-ezno/ - > Introducing Ezno - > Ezno is an experimental compiler I have been working on and off for a while. In short, it is a JavaScript compiler featuring checking, correctness and performance for building full-stack (rendering on the client and server) websites. - https://kaleidawave.github.io/posts/ezno-23/ - > Ezno in '23 - https://github.com/boa-dev/boa - > Boa is an embeddable and experimental Javascript engine written in Rust. Currently, it has support for some of the language. - > This is an experimental Javascript lexer, parser and interpreter written in Rust. Currently, it has support for some of the language. - https://boajs.dev/ - > Boa is an experimental Javascript lexer, parser and compiler written in Rust. Currently, it has support for some of the language. It can be embedded in Rust projects fairly easily and also used from the command line. Boa also exists to serve as a Rust implementation of the EcmaScript specification, there will be areas where we can utilise Rust and its fantastic ecosystem to make a fast, concurrent and safe engine. - https://boajs.dev/boa/playground/ - https://boajs.dev/posts/2022-10-24-boa-usage/ - > Adding a JavaScript interpreter to your Rust project - https://boajs.dev/boa/dev/bench/ - > Boa Benchmarks - https://github.com/boa-dev/boa/blob/main/boa_engine/benches/full.rs#L9 - https://github.com/bheisler/criterion.rs - > Statistics-driven benchmarking library for Rust - > Criterion.rs helps you write fast code by detecting and measuring performance improvements or regressions, even small ones, quickly and accurately. You can optimize with confidence, knowing how each change affects the performance of your code. - https://docs.rs/criterion/latest/criterion/ - https://boajs.dev/boa/test262/ - > EcmaScript conformance test results for Boa - https://github.com/rust-analyzer/rowan - > Rowan is a library for lossless syntax trees, inspired in part by Swift's libsyntax. - https://github.com/mozilla-spidermonkey/jsparagus - > Experimental JS parser-generator project. - > jsparagus - A JavaScript parser written in Rust > jsparagus is intended to replace the JavaScript parser in Firefox. - https://github.com/mozilla-spidermonkey/jsparagus#benchmarking - > Benchmarking - https://github.com/mozilla-spidermonkey/jsparagus#limitations - > Limitations - https://github.com/ratel-rust/ratel-core - > High performance JavaScript to JavaScript compiler with a Rust core - https://maciej.codes/ratel-wasm/ (REPL / Playground) - https://github.com/ratel-rust/ratel-cli - > A command-line interface for Ratel, a high performance JavaScript to JavaScript compiler with a Rust core - https://github.com/dandavison/delta - > A syntax-highlighting pager for git, diff, and grep output - > (the package is called "git-delta" in most package managers, but the executable is just delta) -
0xdevalias revised this gist
Dec 1, 2023 . 1 changed file with 2 additions and 2 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -219,8 +219,8 @@ - https://joern.io/ - > The Bug Hunter's Workbench - > Query: Uncover attack surface, sloppy coding practices, and variants of known vulnerabilities using an interactive code analysis shell. Joern supports C, C++, LLVM bitcode, x86 binaries (via Ghidra), JVM bytecode (via Soot), and Javascript. Python, Java source code, Kotlin, and PHP support coming soon. - > Automate: Wrap your queries into custom code scanners and share them with the community or run existing Joern-based scanners in your CI. - > Integrate: Use Joern as a library to power your own code analysis tools or as a component via the REST API. - https://github.com/joernio/joern - > Open-source code analysis platform for C/C++/Java/Binary/Javascript/Python/Kotlin based on code property graphs. - > Joern is a platform for analyzing source code, bytecode, and binary executables. It generates code property graphs (CPGs), a graph representation of code for cross-language code analysis. Code property graphs are stored in a custom graph database. This allows code to be mined using search queries formulated in a Scala-based domain-specific query language. Joern is developed with the goal of providing a useful tool for vulnerability discovery and research in static program analysis. -
0xdevalias revised this gist
Nov 27, 2023 . 1 changed file with 3 additions and 0 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -158,6 +158,9 @@ > VSCode extension package for ast-grep language server - https://github.com/ast-grep/ast-grep/issues/80 - > Using ast-grep as a library - https://ast-grep.github.io/guide/api-usage/py-api.html - > Python API > `ast-grep`'s Python API is powered by PyO3. You can write Python to programmatically inspect and change syntax trees. - https://github.com/ast-grep/ast-grep/issues/524 - > More powerful `tree-sitter` analysis - https://github.com/microsoft/vscode-anycode -
0xdevalias revised this gist
Nov 27, 2023 . 1 changed file with 4 additions and 0 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -700,6 +700,10 @@ # } #} ``` - https://github.com/lebab/lebab - > Turn your ES5 code into readable ES6. Lebab does the opposite of what Babel does. - > Lebab transpiles your ES5 code to ES6/ES7. It does exactly the opposite of what Babel does. - https://lebab.github.io/ (Playground) ### Recast + related -
0xdevalias revised this gist
Nov 27, 2023 . 1 changed file with 2 additions and 1 deletion.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -1566,4 +1566,5 @@ These are private chat links, so won't work for others, and are included here on - [Reverse Engineered Webpack Tailwind-Styled-Component (0xdevalias gist)](https://gist.github.com/0xdevalias/916e4ababd3cb5e3470b07a024cf3125#reverse-engineered-webpack-tailwind-styled-component) - [Reverse engineering ChatGPT's frontend web app + deep dive explorations of the code (0xdevalias gist)](https://gist.github.com/0xdevalias/4ac297ee3f794c17d0997b4673a2f160#reverse-engineering-chatgpts-frontend-web-app--deep-dive-explorations-of-the-code) - [Bypassing Cloudflare, Akamai, etc (0xdevalias gist)](https://gist.github.com/0xdevalias/b34feb567bd50b37161293694066dd53#bypassing-cloudflare-akamai-etc) - [devalias' Beeper CSS Hacks (0xdevalias gist)](https://gist.github.com/0xdevalias/3d2f5a861335cc1277b21a29d1285cfe#beeper-custom-theme-styles) - [Reverse Engineering Golang (0xdevalias' gist)](https://gist.github.com/0xdevalias/4e430914124c3fd2c51cb7ac2801acba#reverse-engineering-golang) -
0xdevalias revised this gist
Nov 23, 2023 . 1 changed file with 22 additions and 0 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -84,10 +84,32 @@ - https://prettier.io/docs/en/api.html - > API > If you want to run Prettier programmatically, check this page out. - https://github.com/beautify-web/js-beautify - > Beautifier for javascript - > This little beautifier will reformat and re-indent bookmarklets, ugly JavaScript, unpack scripts packed by Dean Edward’s popular packer, as well as partly deobfuscate scripts processed by the npm package `javascript-obfuscator`. - https://beautifier.io/ - https://github.com/shapesecurity/unminify - > Reverse many of the transformations applied by minifiers and naïve obfuscators - https://github.com/shapesecurity/unminify/#safety-levels - https://unminify.io/ - https://github.com/PerimeterX/restringer - > Restringer - > Deobfuscate Javascript and reconstruct strings. Simplify cumbersome logic where possible while adhering to scope limitations. - https://restringer.tech/ (Playground) - https://github.com/PerimeterX/restringer/tree/main/src/processors - > Processors > Processors are a collection of methods meant to prepare the script for obfuscation, removing anti-debugging traps and performing any required modifications before (preprocessors) or after (postprocessors) the main deobfuscation process. > > The processors are created when necessary and are lazily loaded when a specific obfuscation type was detected which requires these additional processes. - https://github.com/PerimeterX/restringer/tree/main/src/modules/safe - https://github.com/PerimeterX/restringer/tree/main/src/modules/unsafe - https://github.com/PerimeterX/obfuscation-detector - > Obfuscation Detector - > Detect different types of JS obfuscation by their AST structure. - https://github.com/PerimeterX/flast - > flAST - FLat Abstract Syntax Tree > Flatten an Abstract Syntax Tree by placing all the nodes in a single flat array. - > Provides a flat Abstract Syntax Tree and an Arborist to trim and modify the tree - https://github.com/lelinhtinh/de4js - > JavaScript Deobfuscator and Unpacker - https://lelinhtinh.github.io/de4js/ -
0xdevalias revised this gist
Nov 23, 2023 . 1 changed file with 42 additions and 0 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -431,6 +431,48 @@ - > Deobfuscating Javascript via AST: Removing Dead or Unreachable Code - https://steakenthusiast.github.io/2022/06/14/Deobfuscating-Javascript-via-AST-Deobfuscating-a-Peculiar-JSFuck-style-Case/ - > Deobfuscating Javascript via AST: A Peculiar JSFuck-esque Case - https://www.trickster.dev/post/ - > Trickster Dev: Code level discussion of web scraping, gray hat automation, growth hacking and bounty hunting - https://www.trickster.dev/post/understanding-abstract-syntax-trees/ - > Understanding Abstract Syntax Trees - https://www.trickster.dev/post/javascript-obfuscation-techniques-by-example/ - > Javascript obfuscation techniques by example - https://www.trickster.dev/post/javascript-ast-manipulation-with-babel-the-first-steps/ - > JavaScript AST manipulation with Babel: the first steps - https://www.trickster.dev/post/javascript-ast-manipulation-with-babel-extracting-hardcoded-data/ - > JavaScript AST manipulation with Babel: extracting hardcoded data - https://www.trickster.dev/post/javascript-ast-manipulation-with-babel-removing-unreachable-code/ - > JavaScript AST manipulation with Babel: removing unreachable code - https://www.trickster.dev/post/javascript-ast-manipulation-with-babel-defeating-string-array-mapping/ - > JavaScript AST manipulation with Babel: defeating string array mapping - https://www.trickster.dev/post/javascript-ast-manipulation-with-babel-transform-prototyping-and-plugin-development/ - > JavaScript AST manipulation with Babel: transform prototyping and plugin development - https://www.trickster.dev/post/javascript-ast-manipulation-with-babel-3-ways-to-create-nodes-and-subtrees/ - > JavaScript AST manipulation with Babel: 3 ways to create nodes and subtrees - https://www.trickster.dev/post/javascript-ast-manipulation-with-babel-untangling-scope-confusion/ - > JavaScript AST manipulation with Babel: untangling scope confusion - https://www.trickster.dev/post/javascript-ast-manipulation-with-babel-ast-modification-apis/ - > JavaScript AST manipulation with Babel: AST modification APIs - https://www.trickster.dev/post/javascript-ast-manipulation-with-babel-constant-folding-and-propagation/ - > JavaScript AST manipulation with Babel: constant folding and propagation - https://www.trickster.dev/post/javascript-ast-manipulation-with-babel-reducing-indirection-undoing-string-concealing/ - > JavaScript AST manipulation with Babel: reducing indirection, undoing string concealing - https://www.trickster.dev/post/javascript-ast-manipulation-with-babel-reducing-nestedness-unflattening-the-cfg/ - > JavaScript AST manipulation with Babel: reducing nestedness, unflattening the CFG - https://www.trickster.dev/post/dont-jsfuck-with-me-part-1/ - > Don’t JSFuck with me: Part 1 - https://www.trickster.dev/post/dont-jsfuck-with-me-part-2/ - > Don’t JSFuck with me: Part 2 - https://www.trickster.dev/post/dont-jsfuck-with-me-part-3/ - > Don’t JSFuck with me: Part 3 - https://www.trickster.dev/post/understanding-javascript-packers/ - > Understanding JavaScript packers - https://www.trickster.dev/post/self-defending-js-code-and-debugger-traps/ - > Self-defending JS code and debugger traps - https://www.trickster.dev/post/restringer-modular-javascript-deobfuscator/ - > Restringer: modular JavaScript deobfuscator - https://www.trickster.dev/post/solving-a-simple-js-challenge-with-sandboxing/ - > Solving a simple JS challenge with sandboxing - https://raz0r.name/articles/using-codeql-to-detect-client-side-vulnerabilities-in-web-applications/ - > Using CodeQL to detect client-side vulnerabilities in web applications -
0xdevalias revised this gist
Nov 23, 2023 . 1 changed file with 39 additions and 0 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -218,6 +218,39 @@ > - Program Dependence graphs (PDG) > - Code Property Graphs (CPG14) > - Entire graph, i.e. convert to a different graph format (ALL) - https://github.com/julianjensen/ast-flow-graph - > ast-flow-graph - > Creates a CFG from JavaScript source code. - > This module will read one or more JavaScript source files and produce CFGs (Control Flow Graphs) of the code. - Uses espree, escope, estraverse, etc - https://github.com/isaacs/yallist - > Yet Another Linked List > > There are many doubly-linked list implementations like it, but this one is mine. > > For when an array would be too big, and a Map can't be iterated in reverse order. - https://github.com/julianjensen/traversals - > Small module for graph traversals, supporting DFS and BFS with niceties added for pre- and post-order, including their reverses. - Some notes from ChatGPT: - > Provides a small module designed for performing graph traversal operations, specifically Depth-First Search (DFS) and Breadth-First Search (BFS). It includes additional features such as pre-order and post-order traversals, as well as their reverse versions, to enhance the functionality of these standard graph traversal techniques. - https://github.com/julianjensen/dominators - > Various dominator tree algorithms - > It implements two different methods for finding the immediate dominators of a graph. - Some notes from ChatGPT: - > A dominator tree is a concept used in computer science, particularly in the field of compiler design and program analysis. To understand a dominator tree, let's first look at the concept of dominators in a control flow graph (CFG). > > In a CFG, which represents the flow of control in a program, a node \( A \) is said to dominate another node \( B \) if every path from the start node of the graph to \( B \) must go through \( A \). In other words, \( A \) dominates \( B \) if \( A \) is always encountered before \( B \) when traversing the graph from the start node. > > The concept becomes more nuanced with the idea of immediate dominators. An immediate dominator of a node \( B \) is the last dominator on any path from the start node to \( B \). > > Now, a dominator tree is a tree structure that represents these dominance relationships within a CFG. In this tree: > > - Each node corresponds to a node in the original CFG. > - There is a directed edge from node \( A \) to node \( B \) if \( A \) is the immediate dominator of \( B \) in the CFG. > > A dominator tree generator, therefore, is a tool or an algorithm that constructs the dominator tree from a given control flow graph. This tool is essential in optimizing compilers and in various program analysis tasks, where understanding the dominance relationships helps in transformations like loop optimization, dead code elimination, and more sophisticated analyses like static single assignment (SSA) form conversion. > > This concept is closely related to computer science and software engineering, particularly in areas concerning compiler construction and code optimization. Given your background in software engineering and ethical hacking, this knowledge could be particularly useful in understanding code structure and flow, especially when analyzing or optimizing complex software systems. ### wakaru @@ -1276,6 +1309,12 @@ In addition to the links directly below, also make sure to check out the various - https://www.hex-rays.com/products/ida/support/idadoc/609.shtml - > IDA Help: Names Representation - > Dummy names are automatically generated by IDA. They are used to denote subroutines, program locations and data. Dummy names have various prefixes depending on the item type and value - https://binary.ninja/2023/09/15/3.5-expanded-universe.html#automatic-variable-naming - > Automatic Variable Naming > One easy way to improve decompilation output is to come up with [better default names](https://github.com/Vector35/binaryninja-api/issues/2558) for variables. There’s a lot of possible defaults you could choose and a number of different strategies are seen throughout different reverse engineering tools. Prior to 3.5, Binary Ninja left variables named based on their origin. Stack variables were var_OFFSET, register-based variables were reg_COUNTER, and global data variables were (data_). While this scheme isn’t changing, we’re being much more intelligent about situations where additional information is available. > > For example, if a variable is passed to a function and a variable name is available, we can now make a much better guess for the variable name. This is most obvious in binaries with type libraries. - > This isn’t the only style of default names. Binary Ninja also will name loop counters with simpler names like `i`, or `j`, `k`, etc (in the case of nested loops) - Webpack - https://webpack.js.org/configuration/optimization/#optimizationchunkids - > `optimization.chunkIds` -
0xdevalias revised this gist
Nov 23, 2023 . 1 changed file with 53 additions and 1 deletion.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -39,6 +39,41 @@ ### Unsorted - https://eslint.org/docs/ - https://eslint.org/docs/latest/extend/custom-rules#the-context-object - > The context object is the only argument of the create method in a rule. - > As the name implies, the context object contains information that is relevant to the context of the rule. - https://eslint.org/docs/latest/extend/custom-rules#applying-fixes - > Applying Fixes > If you’d like ESLint to attempt to fix the problem you’re reporting, you can do so by specifying the fix function when using `context.report()`. The `fix` function receives a single argument, a `fixer` object, that you can use to apply a fix. - > Important: The `meta.fixable` property is mandatory for fixable rules. ESLint will throw an error if a rule that implements `fix` functions does not export the `meta.fixable` property. - > The `fixer` object has the following methods: > > - `insertTextAfter(nodeOrToken, text)`: Insert text after the given node or token. > - `insertTextAfterRange(range, text)`: Insert text after the given range. > - `insertTextBefore(nodeOrToken, text)`: Insert text before the given node or token. > - `insertTextBeforeRange(range, text)`: Insert text before the given range. > - `remove(nodeOrToken)`: Remove the given node or token. > - `removeRange(range)`: Remove text in the given range. > - `replaceText(nodeOrToken, text)`: Replace the text in the given node or token. > - `replaceTextRange(range, text)`: Replace the text in the given range. > > A `range` is a two-item array containing character indices inside the source code. The first item is the start of the range (inclusive) and the second item is the end of the range (exclusive). Every node and token has a `range` property to identify the source code range they represent. > > The above methods return a `fixing` object. The `fix()` function can return the following values: > > - A `fixing` object. > - An array which includes `fixing` objects. > - An iterable object which enumerates `fixing` objects. Especially, the `fix()` function can be a generator. > > If you make a `fix()` function which returns multiple `fixing` objects, those `fixing` objects must not overlap. - https://eslint.org/docs/latest/extend/code-path-analysis - > Code Path Analysis Details - > ESLint’s rules can use code paths. The code path is execution routes of programs. It forks/joins at such as `if` statements. - > Program is expressed with several code paths. A code path is expressed with objects of two kinds: `CodePath` and `CodePathSegment`. - > `CodePath` expresses whole of one code path. This object exists for each function and the global. This has references of both the initial segment and the final segments of a code path. - > `CodePathSegment` is a part of a code path. A code path is expressed with plural `CodePathSegment` objects, it’s similar to doubly linked list. Difference from doubly linked list is what there are forking and merging (the next/prev are plural). - > There are seven events related to code paths, and you can define event handlers by adding them alongside node visitors in the object exported from the `create()` method of your rule. - https://prettier.io/ - https://github.com/prettier/prettier - > Prettier is an opinionated code formatter. It enforces a consistent style by parsing your code and re-printing it with its own rules that take the maximum line length into account, wrapping code when necessary. @@ -688,6 +723,7 @@ - https://github.com/bvaughn/babel-repl - > React powered Babel REPL - https://babeljs.io/docs/babel-parser - > @babel/parser - > The Babel parser (previously Babylon) is a JavaScript parser used in Babel - > Heavily based on `acorn` and `acorn-jsx` - https://babeljs.io/docs/babel-parser#api @@ -720,6 +756,7 @@ functions: Set<string> = new Set(); // A set of lexically-declared FunctionDeclaration names in the current lexical scope` ``` - https://babeljs.io/docs/babel-traverse - > @babel/traverse - > We can use it alongside the `babel` parser to traverse and update nodes - https://github.com/babel/babel/tree/main/packages/babel-traverse - https://github.com/babel/babel/blob/main/packages/babel-traverse/src/traverse-node.ts#L8-L20 @@ -768,6 +805,15 @@ toString() // Generates the code for this path // etc ``` - https://babeljs.io/docs/babel-generator - > @babel/generator - > Turns an AST into code. - https://babeljs.io/docs/babel-template - > @babel/template - > When calling template as a function with a string argument, you can provide placeholders which will get substituted when the template is used. - https://babeljs.io/docs/babel-code-frame - > @babel/code-frame - > Babel Code Frame is a package in the Babel toolchain that generates errors containing a "code frame" which points to specific source locations in the code, aiding in debugging and error analysis - https://babeljs.io/docs/babel-types - https://babeljs.io/docs/babel-types#aliases - https://babeljs.io/docs/babel-types#scopable @@ -776,7 +822,13 @@ - > A cover of AST nodes that start an execution context with new `VariableEnvironment`. In other words, they define the scope of `var` declarations. `FunctionParent` did not include `Program` since Babel 7. - https://babeljs.io/docs/babel-types#blockparent - > A cover of AST nodes that start an execution context with new `LexicalEnvironment`. In other words, they define the scope of `let` and `const` declarations. - https://babeljs.io/docs/babel-helper-module-imports - > @babel/helper-module-imports - https://babeljs.io/docs/babel-helper-validator-identifier - > @babel/helper-validator-identifier is a utility package for parsing JavaScript keywords and identifiers. It provides several helper functions for identifying valid identifier names and detecting reserved words and keywords. - https://babeljs.io/docs/babel-helper-environment-visitor - > @babel/helper-environment-visitor is a utility package that provides a current this context visitor. ### `semantic` / `tree-sitter` + related - https://github.com/github/semantic -
0xdevalias revised this gist
Nov 22, 2023 . 1 changed file with 20 additions and 0 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -1307,6 +1307,26 @@ In addition to the links directly below, also make sure to check out the various > - Symbolic execution and automated theorem provers have limitations on the classes of constraints they can represent and solve. For example, a theorem prover based on linear arithmetic will be unable to cope with the nonlinear path condition xy = 6. Any time that such constraints arise, the symbolic execution may substitute the current concrete value of one of the variables to simplify the problem. An important part of the design of a concolic testing system is selecting a symbolic representation precise enough to represent the constraints of interest. - https://en.wikipedia.org/wiki/Concolic_testing#Tools - Jalangi is an open-source concolic testing and symbolic execution tool for JavaScript. Jalangi supports integers and strings. - https://github.com/Z3Prover/z3 - > The Z3 Theorem Prover - https://github.com/Z3Prover/z3/wiki - > Z3 is an SMT solver and supports the SMTLIB format. - https://smtlib.cs.uiowa.edu/ - > SMT-LIB is an international initiative aimed at facilitating research and development in Satisfiability Modulo Theories (SMT). - > Documents describing the SMT-LIB input/output language for SMT solvers and its semantics; - etc - https://microsoft.github.io/z3guide/ - > Online Z3 Guide - https://github.com/microsoft/z3guide - > Tutorials and courses for Z3 - https://microsoft.github.io/z3guide/docs/logic/intro/ - > Introduction > Z3 is a state-of-the art theorem prover from Microsoft Research. It can be used to check the satisfiability of logical formulas over one or more theories. Z3 offers a compelling match for software analysis and verification tools, since several common software constructs map directly into supported theories. > > The main objective of the tutorial is to introduce the reader on how to use Z3 effectively for logical modeling and solving. The tutorial provides some general background on logical modeling, but we have to defer a full introduction to first-order logic and decision procedures to text-books in order to develop an in depth understanding of the underlying concepts. To clarify: a deep understanding of logical modeling is not necessarily required to understand this tutorial and modeling with Z3, but it is necessary to understand for writing complex models. - https://microsoft.github.io/z3guide/programming/Z3%20JavaScript%20Examples/ - > Z3 JavaScript > The Z3 distribution comes with TypeScript (and therefore JavaScript) bindings for Z3. In the following we give a few examples of using Z3 through these bindings. You can run and modify the examples locally in your browser. - https://github.com/Samsung/jalangi2 - > Dynamic analysis framework for JavaScript - > Jalangi2 is a framework for writing dynamic analyses for JavaScript. Jalangi1 is still available at https://github.com/SRA-SiliconValley/jalangi, but we no longer plan to develop it. Jalangi2 does not support the record/replay feature of Jalangi1. In the Jalangi2 distribution you will find several analyses: -
0xdevalias revised this gist
Nov 22, 2023 . 1 changed file with 33 additions and 5 deletions.There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode charactersOriginal file line number Diff line number Diff line change @@ -268,6 +268,9 @@ > To make the decompilation process more effective and adaptable to different React websites, I recommend a more dynamic approach by identifying the React library being used in the compiled code, instead of hardcoding the use of 'React'. This can possibly be achieved by finding the variable name assigned to the React library and using that in the matchers. - https://github.com/j4k0xb/webcrack/issues/6 - > `(0, fn)(...args)` type of calls - https://github.com/j4k0xb/webcrack/issues/24 - > optimisation ideas - > There are some js to c/wasm transpilers but - https://github.com/e9x/krunker-decompiler - > Krunker Decompiler - > Powered by webcrack @@ -916,6 +919,18 @@ - https://github.com/rustwasm/wasm-pack - > This tool seeks to be a one-stop shop for building and working with rust- generated WebAssembly that you would like to interop with JavaScript, in the browser or with Node.js. `wasm-pack` helps you build rust-generated WebAssembly packages that you could publish to the npm registry, or otherwise use alongside any javascript packages in workflows that you already use - https://rustwasm.github.io/wasm-pack/book/ - https://github.com/coderaiser/swc-to-babel - > convert SWC to Babel AST - https://github.com/coderaiser/estree-to-babel - > convert estree ast to babel - > Convert ESTree-compatible JavaScript AST to Babel AST. - https://github.com/j4k0xb/estree-to-babel/tree/perf - https://github.com/coderaiser/estree-to-babel/compare/master...j4k0xb:estree-to-babel:perf - Replaces cherow with meriyah - > When using [`meriyah`](https://github.com/meriyah/meriyah) and `estree-to-babel`, its up to 2.8x faster than [`@babel/parser`](https://babeljs.io/docs/babel-parser) alone. - https://github.com/meriyah/meriyah - > 100% compliant, self-hosted javascript parser with high focus on both performance and stability. Stable and already used in production. - https://meriyah.github.io/meriyah/ (Playground) ### `esbuild` @@ -1204,6 +1219,11 @@ In addition to the links directly below, also make sure to check out the various ### Variable Name Mangling - https://hex-rays.com/blog/igors-tip-of-the-week-34-dummy-names/ - > In IDA’s disassembly, you may have often observed names that may look strange and cryptic on first sight: `sub_73906D75`, `loc_40721B`, `off_40A27C` and more. In IDA’s terminology, they’re called dummy names. They are used when a name is required by the assembly syntax but there is nothing suitable available - https://www.hex-rays.com/products/ida/support/idadoc/609.shtml - > IDA Help: Names Representation - > Dummy names are automatically generated by IDA. They are used to denote subroutines, program locations and data. Dummy names have various prefixes depending on the item type and value - Webpack - https://webpack.js.org/configuration/optimization/#optimizationchunkids - > `optimization.chunkIds` @@ -1232,8 +1252,6 @@ In addition to the links directly below, also make sure to check out the various - https://webpack.js.org/plugins/terser-webpack-plugin/#terseroptions - `keep_classnames`, `keep_fnames`, `mangle`, `nameCache`, etc - https://github.com/terser/terser - https://webpack.js.org/configuration/optimization/#optimizationminimizer - > `optimization.minimizer` - > Allows you to override the default minimizer by providing a different one or more customized `TerserPlugin` instances. @@ -1307,7 +1325,6 @@ In addition to the links directly below, also make sure to check out the various - > Analysis in node.js with on-the-fly instrumentation - > Analysis in node.js with explicit one-file-at-a-time offline instrumentation - > Analysis in a browser using a proxy and on-the-fly instrumentation - https://github.com/SRA-SiliconValley/jalangi - > This repository has been archived by the owner on Dec 9, 2017. It is now read-only. - > We encourage you to switch to Jalangi2 available at https://github.com/Samsung/jalangi2. Jalangi2 is a framework for writing dynamic analyses for JavaScript. Jalangi2 does not support the record/replay feature of Jalangi1. Jalangi1 is still available from this website, but we no longer plan to develop it. @@ -1344,8 +1361,19 @@ In addition to the links directly below, also make sure to check out the various ## Unsorted - https://github.com/bytecodealliance/ComponentizeJS - > ESM -> WebAssembly Component creator, via a SpiderMonkey JS engine embedding - > Provides a Mozilla SpiderMonkey embedding that takes as input a JavaScript source file and a WebAssembly Component WIT World, and outputs a WebAssembly Component binary with the same interface. - https://bytecodealliance.org/articles/making-javascript-run-fast-on-webassembly - > Making JavaScript run fast on WebAssembly - > We should be clear here—if you’re running JavaScript in the browser, it still makes the most sense to simply deploy JS. The JS engines within the browsers are highly tuned to run the JS that gets shipped to them. - https://github.com/bytecodealliance/wizer - > The WebAssembly Pre-Initializer > Don't wait for your Wasm module to initialize itself, pre-initialize it! Wizer instantiates your WebAssembly module, executes its initialization function, and then snapshots the initialized state out into a new WebAssembly module. Now you can use this new, pre-initialized WebAssembly module to hit the ground running, without making your users wait for that first-time set up code to complete. > > The improvements to start up latency you can expect will depend on how much initialization work your WebAssembly module needs to do before it's ready. Some initial benchmarking shows between 1.35 to 6.00 times faster instantiation and initialization with Wizer, depending on the workload - https://wingolog.org/archives/2022/08/18/just-in-time-code-generation-within-webassembly - > just-in-time code generation within webassembly ## My ChatGPT Research / Conversations
NewerOlder