From 9d20d2281f4698024b7be67d1b86178b4e8e2484 Mon Sep 17 00:00:00 2001 From: Clawd Date: Mon, 16 Feb 2026 12:12:37 -0800 Subject: Clean up project structure - Add README.md - Move benchmark files to benchmarks/ - Move PLAN.md to .claude/ - Add .gitignore --- benchmarks/BENCHMARKS.md | 130 +++++++++++++++++++++++++++++++++ benchmarks/BENCHMARK_SUMMARY.md | 154 +++++++++++++++++++++++++++++++++++++++ benchmarks/benchmark_results.txt | 32 ++++++++ benchmarks/run_benchmarks.sh | 35 +++++++++ 4 files changed, 351 insertions(+) create mode 100644 benchmarks/BENCHMARKS.md create mode 100644 benchmarks/BENCHMARK_SUMMARY.md create mode 100644 benchmarks/benchmark_results.txt create mode 100755 benchmarks/run_benchmarks.sh (limited to 'benchmarks') diff --git a/benchmarks/BENCHMARKS.md b/benchmarks/BENCHMARKS.md new file mode 100644 index 0000000..1a671c1 --- /dev/null +++ b/benchmarks/BENCHMARKS.md @@ -0,0 +1,130 @@ +# Nostr Library Benchmarks + +This directory contains comprehensive benchmarks comparing three popular Go Nostr libraries: + +- **NWIO** (`code.northwest.io/nostr`) - This library +- **NBD** (`github.com/nbd-wtf/go-nostr`) - Popular community library +- **Fiat** (`fiatjaf.com/nostr`) - Original implementation by Fiatjaf + +## Benchmark Categories + +### Event Operations +- **Unmarshal**: Parsing JSON into Event struct +- **Marshal**: Serializing Event struct to JSON +- **Serialize**: Canonical serialization for ID computation +- **ComputeID**: Computing event ID hash +- **Sign**: Signing events with private key +- **Verify**: Verifying event signatures + +### Key Operations +- **GenerateKey**: Generating new private keys + +### Filter Operations +- **FilterMatch**: Simple filter matching (kind, author) +- **FilterMatchComplex**: Complex filter matching (with tags, prefix matching) + +## Running Benchmarks + +**Important**: The comparison benchmarks are in a separate module (`benchmarks/comparison/`) to avoid polluting the main module dependencies with competitor libraries. + +### Quick Start + +Run all comparison benchmarks: +```bash +cd benchmarks/comparison +go test -bench=. -benchmem -benchtime=1s +``` + +### Specific Benchmark Groups + +Event unmarshaling: +```bash +cd benchmarks/comparison +go test -bench=BenchmarkEventUnmarshal -benchmem +``` + +Event signing: +```bash +cd benchmarks/comparison +go test -bench=BenchmarkEventSign -benchmem +``` + +Event verification: +```bash +cd benchmarks/comparison +go test -bench=BenchmarkEventVerify -benchmem +``` + +Filter matching: +```bash +cd benchmarks/comparison +go test -bench=BenchmarkFilterMatch -benchmem +``` + +### Compare Single Library + +NWIO only: +```bash +cd benchmarks/comparison +go test -bench='.*_NWIO' -benchmem +``` + +NBD only: +```bash +cd benchmarks/comparison +go test -bench='.*_NBD' -benchmem +``` + +Fiat only: +```bash +cd benchmarks/comparison +go test -bench='.*_Fiat' -benchmem +``` + +## Analyzing Results + +Use `benchstat` for statistical analysis: + +```bash +# Install benchstat +go install golang.org/x/perf/cmd/benchstat@latest + +# Run benchmarks multiple times and compare +cd benchmarks/comparison +go test -bench=. -benchmem -count=10 > results.txt +benchstat results.txt +``` + +Compare two specific libraries: +```bash +cd benchmarks/comparison +go test -bench='.*_NWIO' -benchmem -count=10 > nwio.txt +go test -bench='.*_NBD' -benchmem -count=10 > nbd.txt +benchstat nwio.txt nbd.txt +``` + +## Understanding the Output + +Example output: +``` +BenchmarkEventSign_NWIO-24 50000 35421 ns/op 1024 B/op 12 allocs/op +``` + +- `50000`: Number of iterations +- `35421 ns/op`: Nanoseconds per operation (lower is better) +- `1024 B/op`: Bytes allocated per operation (lower is better) +- `12 allocs/op`: Memory allocations per operation (lower is better) + +## Performance Tips + +1. **Event Unmarshaling**: Critical for relay implementations +2. **Event Signing**: Important for client implementations +3. **Event Verification**: Critical for all implementations +4. **Filter Matching**: Important for relay implementations with many subscriptions + +## Notes + +- All benchmarks use realistic event data +- Benchmarks run with default Go test timeout +- Results may vary based on hardware and system load +- Use `-benchtime=5s` for more stable results on noisy systems diff --git a/benchmarks/BENCHMARK_SUMMARY.md b/benchmarks/BENCHMARK_SUMMARY.md new file mode 100644 index 0000000..2e087a3 --- /dev/null +++ b/benchmarks/BENCHMARK_SUMMARY.md @@ -0,0 +1,154 @@ +# Benchmark Results Summary + +Comparison of three Go Nostr libraries: **NWIO** (code.northwest.io/nostr), **NBD** (github.com/nbd-wtf/go-nostr), and **Fiat** (fiatjaf.com/nostr) + +## Quick Performance Overview + +### 🏆 Winners by Category + +| Operation | Winner | Performance | +|-----------|--------|-------------| +| **Event Unmarshal** | NWIO/Fiat | ~2.5 µs (tied) | +| **Event Marshal** | NWIO | 1.79 µs (fastest, least memory) | +| **Event Serialize** | NBD | 129 ns (3x faster than NWIO) | +| **Compute ID** | Fiat | 276 ns (2x faster than NWIO) | +| **Generate Key** | NBD | 470 ns (80x faster!) | +| **Event Sign** | NBD/Fiat | ~59 µs (2x faster than NWIO) | +| **Event Verify** | NWIO | 99.7 µs (slightly faster) | +| **Filter Match** | NWIO | 7.1 ns (2x faster than Fiat) | +| **Filter Complex** | NWIO | 30.9 ns (fastest) | + +## Detailed Results + +### Event Unmarshaling (JSON → Event) +``` +NWIO: 2,541 ns/op 888 B/op 17 allocs/op ⭐ FASTEST, LOW MEMORY +NBD: 2,832 ns/op 944 B/op 13 allocs/op +Fiat: 2,545 ns/op 752 B/op 10 allocs/op ⭐ LEAST MEMORY +``` +**Analysis**: All three are very competitive. NWIO and Fiat are effectively tied. Fiat uses least memory. + +### Event Marshaling (Event → JSON) +``` +NWIO: 1,790 ns/op 1,010 B/op 3 allocs/op ⭐ FASTEST, LEAST ALLOCS +NBD: 1,819 ns/op 1,500 B/op 6 allocs/op +Fiat: 1,971 ns/op 2,254 B/op 13 allocs/op +``` +**Analysis**: NWIO is fastest with minimal allocations. Significant memory advantage over competitors. + +### Event Serialization (for ID computation) +``` +NWIO: 391 ns/op 360 B/op 7 allocs/op +NBD: 129 ns/op 208 B/op 2 allocs/op ⭐ FASTEST, 3x faster +Fiat: 161 ns/op 400 B/op 3 allocs/op +``` +**Analysis**: NBD dominates here with optimized serialization. NWIO has room for improvement. + +### Event ID Computation +``` +NWIO: 608 ns/op 488 B/op 9 allocs/op +NBD: 302 ns/op 336 B/op 4 allocs/op +Fiat: 276 ns/op 400 B/op 3 allocs/op ⭐ FASTEST +``` +**Analysis**: NBD and Fiat are 2x faster. NWIO should optimize ID computation path. + +### Key Generation +``` +NWIO: 37,689 ns/op 208 B/op 4 allocs/op +NBD: 470 ns/op 369 B/op 8 allocs/op ⭐ FASTEST, 80x faster! +Fiat: 25,375 ns/op 272 B/op 5 allocs/op +``` +**Analysis**: ⚠️ NWIO is significantly slower. NBD appears to use a different key generation strategy. This is the biggest performance gap. + +### Event Signing +``` +NWIO: 129,854 ns/op 2,363 B/op 42 allocs/op +NBD: 59,069 ns/op 2,112 B/op 35 allocs/op ⭐ TIED FASTEST +Fiat: 58,572 ns/op 1,760 B/op 29 allocs/op ⭐ LEAST MEMORY +``` +**Analysis**: NBD and Fiat are 2x faster. NWIO has more allocations in signing path. + +### Event Verification +``` +NWIO: 99,744 ns/op 953 B/op 19 allocs/op ⭐ FASTEST +NBD: 105,995 ns/op 624 B/op 11 allocs/op ⭐ LEAST MEMORY +Fiat: 103,744 ns/op 640 B/op 9 allocs/op +``` +**Analysis**: NWIO is slightly faster (6% faster than others). Very competitive across all three. + +### Filter Matching (Simple) +``` +NWIO: 7.1 ns/op 0 B/op 0 allocs/op ⭐ FASTEST, 2x faster +NBD: 10.8 ns/op 0 B/op 0 allocs/op +Fiat: 16.4 ns/op 0 B/op 0 allocs/op +``` +**Analysis**: NWIO excels at filter matching! Zero allocations across all libraries. + +### Filter Matching (Complex with Tags) +``` +NWIO: 30.9 ns/op 0 B/op 0 allocs/op ⭐ FASTEST +NBD: 33.4 ns/op 0 B/op 0 allocs/op +Fiat: 42.6 ns/op 0 B/op 0 allocs/op +``` +**Analysis**: NWIO maintains lead in complex filters. Important for relay implementations. + +## Optimization Opportunities for NWIO + +### High Priority 🔴 +1. **Key Generation** - 80x slower than NBD + - Current: 37.7 µs + - Target: ~500 ns (similar to NBD) + - Impact: Critical for client applications + +2. **Event Signing** - 2x slower than competitors + - Current: 130 µs + - Target: ~60 µs (match NBD/Fiat) + - Impact: High for client applications + +### Medium Priority 🟡 +3. **Event Serialization** - 3x slower than NBD + - Current: 391 ns + - Target: ~130 ns (match NBD) + - Impact: Used in ID computation + +4. **ID Computation** - 2x slower than competitors + - Current: 608 ns + - Target: ~280 ns (match Fiat) + - Impact: Affects every event processing + +## Current Strengths of NWIO ✅ + +1. **Filter Matching** - 2x faster than Fiat, fastest overall +2. **Event Marshaling** - Fastest with minimal allocations +3. **Event Verification** - Slightly faster than competitors +4. **Memory Efficiency** - Competitive or better in most operations + +## Recommendations + +### For Relay Implementations +- **NWIO excels**: Best filter matching performance +- All three are competitive for event parsing/verification + +### For Client Implementations +- **NBD/Fiat preferred**: Much faster key generation and signing +- NWIO needs optimization in crypto operations + +### Overall Assessment +- **NWIO**: Best for relay/filter-heavy workloads +- **NBD**: Most balanced, excellent crypto performance +- **Fiat**: Good all-around, lowest memory in some operations + +## Running Your Own Benchmarks + +```bash +# Run all benchmarks +./run_benchmarks.sh + +# Compare specific operations +go test -bench=BenchmarkEventSign -benchmem comparison_bench_test.go +go test -bench=BenchmarkFilterMatch -benchmem comparison_bench_test.go + +# Statistical analysis with benchstat +go test -bench=. -count=10 comparison_bench_test.go > results.txt +benchstat results.txt +``` diff --git a/benchmarks/benchmark_results.txt b/benchmarks/benchmark_results.txt new file mode 100644 index 0000000..c3976e6 --- /dev/null +++ b/benchmarks/benchmark_results.txt @@ -0,0 +1,32 @@ +goos: linux +goarch: amd64 +cpu: AMD Ryzen AI 9 HX PRO 370 w/ Radeon 890M +BenchmarkEventUnmarshal_NWIO-24 498826 2541 ns/op 888 B/op 17 allocs/op +BenchmarkEventUnmarshal_NBD-24 423019 2832 ns/op 944 B/op 13 allocs/op +BenchmarkEventUnmarshal_Fiat-24 430042 2545 ns/op 752 B/op 10 allocs/op +BenchmarkEventMarshal_NWIO-24 613165 1790 ns/op 1010 B/op 3 allocs/op +BenchmarkEventMarshal_NBD-24 620986 1819 ns/op 1500 B/op 6 allocs/op +BenchmarkEventMarshal_Fiat-24 621964 1971 ns/op 2254 B/op 13 allocs/op +BenchmarkEventSerialize_NWIO-24 3059661 391.0 ns/op 360 B/op 7 allocs/op +BenchmarkEventSerialize_NBD-24 8824029 128.8 ns/op 208 B/op 2 allocs/op +BenchmarkEventSerialize_Fiat-24 6533536 160.9 ns/op 400 B/op 3 allocs/op +BenchmarkComputeID_NWIO-24 2108437 608.0 ns/op 488 B/op 9 allocs/op +BenchmarkComputeID_NBD-24 4072243 302.2 ns/op 336 B/op 4 allocs/op +BenchmarkComputeID_Fiat-24 4421660 275.9 ns/op 400 B/op 3 allocs/op +BenchmarkGenerateKey_NWIO-24 31942 37689 ns/op 208 B/op 4 allocs/op +BenchmarkGenerateKey_NBD-24 2489169 469.6 ns/op 369 B/op 8 allocs/op +BenchmarkGenerateKey_Fiat-24 45475 25375 ns/op 272 B/op 5 allocs/op +BenchmarkEventSign_NWIO-24 9072 129854 ns/op 2363 B/op 42 allocs/op +BenchmarkEventSign_NBD-24 20325 59069 ns/op 2112 B/op 35 allocs/op +BenchmarkEventSign_Fiat-24 20613 58572 ns/op 1760 B/op 29 allocs/op +BenchmarkEventVerify_NWIO-24 12009 99744 ns/op 953 B/op 19 allocs/op +BenchmarkEventVerify_NBD-24 10000 105995 ns/op 624 B/op 11 allocs/op +BenchmarkEventVerify_Fiat-24 10000 103744 ns/op 640 B/op 9 allocs/op +BenchmarkFilterMatch_NWIO-24 167376669 7.091 ns/op 0 B/op 0 allocs/op +BenchmarkFilterMatch_NBD-24 100000000 10.82 ns/op 0 B/op 0 allocs/op +BenchmarkFilterMatch_Fiat-24 71761591 16.40 ns/op 0 B/op 0 allocs/op +BenchmarkFilterMatchComplex_NWIO-24 39214178 30.88 ns/op 0 B/op 0 allocs/op +BenchmarkFilterMatchComplex_NBD-24 35580048 33.40 ns/op 0 B/op 0 allocs/op +BenchmarkFilterMatchComplex_Fiat-24 28026481 42.64 ns/op 0 B/op 0 allocs/op +PASS +ok command-line-arguments 40.651s diff --git a/benchmarks/run_benchmarks.sh b/benchmarks/run_benchmarks.sh new file mode 100755 index 0000000..87dfd17 --- /dev/null +++ b/benchmarks/run_benchmarks.sh @@ -0,0 +1,35 @@ +#!/bin/bash +set -e + +# Colors for output +GREEN='\033[0;32m' +BLUE='\033[0;34m' +YELLOW='\033[1;33m' +NC='\033[0m' # No Color + +echo -e "${BLUE}Running Nostr Library Benchmarks${NC}" +echo -e "${BLUE}Comparing: NWIO vs NBD-WTF vs Fiatjaf${NC}" +echo "" + +# First, install comparison dependencies if needed +echo -e "${YELLOW}Ensuring comparison dependencies are available...${NC}" +go get -tags=benchcmp -t ./... +echo "" + +# Run all benchmarks with the benchcmp build tag +echo -e "${GREEN}Running all benchmarks...${NC}" +go test -tags=benchcmp -bench=. -benchmem -benchtime=1s -run=^$ | tee benchmark_results.txt + +echo "" +echo -e "${GREEN}Results saved to benchmark_results.txt${NC}" +echo "" +echo -e "${BLUE}To run specific benchmark groups:${NC}" +echo " go test -tags=benchcmp -bench=BenchmarkEventUnmarshal -benchmem" +echo " go test -tags=benchcmp -bench=BenchmarkEventSign -benchmem" +echo " go test -tags=benchcmp -bench=BenchmarkEventVerify -benchmem" +echo " go test -tags=benchcmp -bench=BenchmarkFilterMatch -benchmem" +echo "" +echo -e "${BLUE}To compare specific libraries:${NC}" +echo " go test -tags=benchcmp -bench='.*_NWIO' -benchmem" +echo " go test -tags=benchcmp -bench='.*_NBD' -benchmem" +echo " go test -tags=benchcmp -bench='.*_Fiat' -benchmem" -- cgit v1.2.3