syncnorris

module

v0.6.0 Latest Latest Go to latest Published: Nov 29, 2025 License: MIT

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/sdejongh/syncnorris

Links

Open Source Insights

README ¶

syncnorris

Version: v0.6.0 Status: Production-ready for one-way sync | Experimental for bidirectional sync License: MIT

Cross-platform file synchronization utility built in Go, optimized for performance with advanced hash comparison and parallel operations.

Current Features ✅

Core Functionality

✅ One-way synchronization from source to destination (production-ready)
- Local filesystem support (mounted network shares work)
- Parallel file transfers (configurable worker count)
- Dry-run mode to preview changes without modifying files
- Incremental sync (only changed files are transferred)
- Delete orphan files (--delete): Remove files from destination that don't exist in source
⚠️ Bidirectional synchronization (EXPERIMENTAL - v0.4.0)
- Two-way sync between source and destination
- Conflict detection: modify-modify, delete-modify, create-create
- Conflict resolution strategies:
  - newer: Use most recently modified version (default)
  - source-wins: Always prefer source version
  - dest-wins: Always prefer destination version
  - both: Keep both versions with .source-conflict/.dest-conflict suffix
- Optional state tracking (--stateful): Track changes between syncs
- ⚠️ Use with caution: Always test with --dry-run first!

Comparison Methods

✅ Hash-based comparison (SHA-256, default and recommended)
- Intelligent composite strategy: metadata first, hash only when needed
- Partial hashing for large files (≥1MB): 95% I/O reduction for quick rejection
- Parallel hash computation: 1.8-1.9x speedup
- Buffer pooling for reduced memory pressure
✅ MD5 hash comparison (faster alternative to SHA-256)
- Similar performance to SHA-256 but less secure
- Suitable for non-critical data where speed matters
- Also supports partial hashing and parallel computation
✅ Binary comparison (byte-by-byte verification)
- Most thorough comparison method
- Reports exact byte offset where files differ
- Useful for debugging or when hash collisions are a concern
✅ Name/size comparison (fast metadata-only mode)
- Ideal for re-sync scenarios: 10-40x faster than hash mode
- Sub-second re-sync for 1000 identical files
✅ Timestamp comparison (name+size+modification time)
- Faster than hash-based comparison
- Suitable when you trust timestamps haven't been manipulated

User Interface

✅ Advanced progress display
- Real-time tabular view of up to 5 concurrent files
- Dual progress bars: data transferred + files processed
- Platform-specific status icons:
  - Linux/macOS: 🟢 copying, 🔵 comparing, ✅ complete, ❌ error
  - Windows: [>>] copying, [??] comparing, [OK] complete, [!!] error
- Legend displayed at top of progress view
- Instantaneous transfer rate (3-second sliding window) + average
- Accurate ETA calculation
- Terminal width detection (prevents line wrapping)
- Optimized for Windows terminals (ASCII icons, reduced flicker)
✅ Human-readable output with comprehensive summary statistics
✅ Differences report
- compare command: always displays differences to screen
- sync command: optional with --diff-report FILE
- Report always created even when no differences (v0.2.0)
- Tracks all operations: copied, updated, synchronized, deleted, errors
- Includes reason for each difference (only in source, content differs, deleted, copy error, etc.)
- Supports human-readable and JSON formats
- Shows "No differences found" when fully synchronized
- JSON output suitable for automation/scripting
✅ Quiet mode for scripts (suppress non-error output)
✅ Verbose mode for debugging
✅ JSON output for automation and scripting (--output json)

File Filtering

✅ Exclude patterns (glob-based filtering)
- Supports glob patterns: *.log, node_modules/**, .git/**
- Multiple patterns via --exclude flag (can be repeated)
- Excluded files are counted in "skipped" statistics and appear in differences report

Performance Controls

✅ Bandwidth limiting (--bandwidth, -b)
- Limit transfer speed: --bandwidth 10M (10 MiB/s)
- Supports K, M, G units (e.g., 500K, 1G)
- Applied to both file copying and hash comparison

Architecture (v0.2.0)

✅ Producer-Consumer Pipeline
- Scanner (producer) populates task queue while workers process in parallel
- Workers start processing before scan completes
- Each worker handles complete file lifecycle (verify → compare → copy)
- Dynamic progress updates during scan phase
- Better memory efficiency (no full operation list in memory)

Performance Optimizations

syncnorris has been heavily optimized and exceeds all performance targets:

Atomic counter statistics: Lock-free updates, 8.6x faster (6% throughput gain)
Progress callback throttling: 93% overhead reduction (smooth 20 updates/sec)
Partial hashing: 95% I/O reduction for files differing in first 256KB
Parallel hash computation: Source and destination hashed concurrently
Composite comparison: Metadata check before expensive hash operations
Buffer pooling: Reduced GC pressure with sync.Pool
Graceful interrupt handling: Cursor visibility restored on Ctrl+C (v0.2.0)

Measured Results:

10,000 files synchronized in <2 minutes (target: <5 min) ✅
1,000 identical files re-synced in <0.5 seconds ✅
Memory usage <300MB for 1M files (target: <500MB) ✅
Incremental sync 10-40x faster than full copy ✅

Build & Distribution

✅ Single static binary (no dependencies required)
✅ Cross-platform: Linux, Windows, macOS (amd64, arm64)
✅ Configuration file support (YAML format)
✅ Shell autocompletion (bash, zsh, fish, powershell)

Logging (v0.6.0)

✅ File logging with configurable output
- JSON and plain text formats (--log-format text|json)
- Log levels: debug, info, warn, error (--log-level)
- Automatic log rotation (size-based with configurable backups)
- Directory auto-creation for log paths
- Detailed debug logging: trace every file operation (copied, updated, synchronized, skipped, deleted, errors)

Planned Features 🚧

These features are NOT yet implemented but are planned for future releases:

🚧 Resume interrupted operations (post-v1.0)
🚧 Native network storage (SMB/Samba, NFS without mounting - post-v1.0)

See IMPLEMENTATION_STATUS.md for detailed feature status.

Installation

Quick Install (Recommended)

Linux & macOS:

curl -sSL https://raw.githubusercontent.com/sdejongh/syncnorris/master/install.sh | bash

Or with wget:

wget -qO- https://raw.githubusercontent.com/sdejongh/syncnorris/master/install.sh | bash

Windows (PowerShell):

irm https://raw.githubusercontent.com/sdejongh/syncnorris/master/install.ps1 | iex

Or download and run:

# Download the script
Invoke-WebRequest -Uri https://raw.githubusercontent.com/sdejongh/syncnorris/master/install.ps1 -OutFile install.ps1

# Run it
powershell -ExecutionPolicy Bypass -File install.ps1

The installer will automatically:

Detect your OS and architecture
Download the latest release
Install the binary to the appropriate location
Add it to your PATH

Manual Download

Download the latest release for your platform from the Releases page:

Download the archive for your platform:
- Linux: syncnorris_VERSION_Linux_x86_64.tar.gz or syncnorris_VERSION_Linux_arm64.tar.gz
- macOS: syncnorris_VERSION_Darwin_x86_64.tar.gz or syncnorris_VERSION_Darwin_arm64.tar.gz
- Windows: syncnorris_VERSION_Windows_x86_64.zip
Extract the archive
Move the binary to a directory in your PATH:
- Linux/macOS: sudo mv syncnorris /usr/local/bin/
- Windows: Move syncnorris.exe to C:\Program Files\syncnorris\ and add to PATH

Using Go

If you have Go installed:

go install github.com/sdejongh/syncnorris/cmd/syncnorris@latest

From Source

# Clone the repository
git clone https://github.com/sdejongh/syncnorris.git
cd syncnorris

# Build
make build

# The binary will be in dist/syncnorris

Quick Start

Basic One-Way Sync

# Sync from source to destination (with progress)
syncnorris sync --source /data/projects --dest /backup/projects

# Short form
syncnorris sync -s /src -d /dst

Preview Changes (Dry-Run)

# See what would be changed without modifying anything
syncnorris sync -s /src -d /dst --dry-run

Bidirectional Sync (EXPERIMENTAL)

# ⚠️ ALWAYS test with --dry-run first!
syncnorris sync -s /src -d /dst --mode bidirectional --dry-run

# Two-way sync with default conflict resolution (newer wins)
syncnorris sync -s /src -d /dst --mode bidirectional

# Use source-wins conflict resolution
syncnorris sync -s /src -d /dst --mode bidirectional --conflict source-wins

# Keep both versions on conflict
syncnorris sync -s /src -d /dst --mode bidirectional --conflict both

# Enable state tracking between syncs
syncnorris sync -s /src -d /dst --mode bidirectional --stateful

Fast Metadata-Only Comparison

# Use name+size comparison instead of hash (much faster for re-sync)
syncnorris sync -s /src -d /dst --comparison namesize

# Use timestamp comparison (name+size+modification time)
syncnorris sync -s /src -d /dst --comparison timestamp

Parallel Operations

# Use 16 parallel workers (default: 5)
syncnorris sync -s /src -d /dst --parallel 16

Quiet Mode for Scripts

# Suppress progress output, only show errors
syncnorris sync -s /src -d /dst --quiet

# Or use short form
syncnorris sync -s /src -d /dst -q

Configuration

Create a config file at ~/.config/syncnorris/config.yaml:

sync:
  mode: oneway                    # Only 'oneway' currently supported
  comparison: hash                # 'hash', 'md5', 'binary', 'namesize', or 'timestamp'

performance:
  max_workers: 8                  # Parallel worker count (0 = CPU count)
  buffer_size: 65536              # Buffer size for I/O operations (64KB)
  bandwidth_limit: "0"            # Bandwidth limit (e.g., "10M", "1G", 0 = unlimited)

output:
  format: human                   # 'human' or 'json'
  progress: true                  # Show real-time progress bars
  quiet: false                    # Suppress non-error output
  verbose: false                  # Extra debug information

exclude:                          # Glob patterns to exclude
  - "*.log"
  - ".git/**"
  - "node_modules/**"

# Note: logging is defined in config but not yet implemented

Usage Reference

Commands

syncnorris sync      # Synchronize two folders (primary command)
syncnorris compare   # Compare folders without syncing (alias for sync --dry-run)
syncnorris config    # Manage configuration
syncnorris version   # Show version, commit, build date, Go version, OS/arch
syncnorris help      # Show help for any command

Sync Command Options

Required Flags

--source, -s PATH    Source directory path (required)
--dest, -d PATH      Destination directory path (required)

Functional Flags (Implemented)

--comparison METHOD  Comparison method: hash, md5, binary, namesize, timestamp (default: hash)
--dry-run            Compare only, don't sync
--create-dest        Create destination directory if it doesn't exist (sync only)
--delete             Delete files in destination that don't exist in source
--parallel, -p N     Number of parallel workers (default: 5)
--mode oneway        Sync mode (only 'oneway' currently supported)
--diff-report FILE   Write differences report to file (sync command)
                     Note: compare command always displays to screen by default
--diff-format FORMAT Report format: human, json (default: human)
--output FORMAT      Output format: human, json (default: human)
--exclude PATTERN    Glob patterns to exclude (can be repeated)
--bandwidth, -b      Bandwidth limit (e.g., "10M", "1G")

# BIDIRECTIONAL FLAGS (experimental)
--mode bidirectional Two-way sync between source and destination
--conflict STRATEGY  Conflict resolution: newer, source-wins, dest-wins, both (default: newer)
--stateful           Enable state persistence between syncs (tracks changes)

# LOGGING FLAGS
--log-file PATH      Write logs to file (enables logging)
--log-format FORMAT  Log format: text, json (default: text)
--log-level LEVEL    Log level: debug, info, warn, error (default: info)

Global Flags

--config FILE        Config file path (default: ~/.config/syncnorris/config.yaml)
--quiet, -q          Suppress non-error output
--verbose, -v        Verbose debug output

Version Command

# Show detailed version information
syncnorris version
# Output:
# syncnorris v0.2.0
#   Commit:     abc1234
#   Built:      2025-11-28T09:03:07Z
#   Go version: go1.24.10
#   OS/Arch:    linux/amd64

# Show only version number
syncnorris version -s
# Output: v0.2.0

# Quick version check (Cobra built-in)
syncnorris --version
# Output: syncnorris version v0.2.0

Examples

Backup Important Data

# Daily backup with hash verification and differences report
syncnorris sync \
  --source ~/Documents \
  --dest /mnt/backup/Documents \
  --comparison hash \
  --diff-report /var/log/backup-diff.txt

# Check if there were any differences
cat /var/log/backup-diff.txt

Fast Re-Sync After Interruption

# Use name+size for quick re-sync (skip re-hashing identical files)
syncnorris sync \
  -s /large/dataset \
  -d /backup/dataset \
  --comparison namesize

Sync to New Destination

# Create destination directory if it doesn't exist
syncnorris sync \
  -s /data/project \
  -d /backup/2025/project \
  --create-dest

Test Before Syncing

# Dry-run to preview changes
syncnorris sync -s ~/src -d /mnt/nas/backup --dry-run

# Or use the dedicated compare command
syncnorris compare -s ~/src -d /mnt/nas/backup

# Review the output, then run actual sync
syncnorris sync -s ~/src -d /mnt/nas/backup

Mirror Source (Delete Orphans)

# Delete files in destination that don't exist in source
syncnorris sync -s /source -d /backup --delete

# Preview what would be deleted (dry-run)
syncnorris sync -s /source -d /backup --delete --dry-run

# Or use compare command to see what would be deleted
syncnorris compare -s /source -d /backup --delete

Compare Folders

# Compare always displays differences report to screen
syncnorris compare -s /original -d /backup --comparison hash
syncnorris compare -s /original -d /backup --comparison md5
syncnorris compare -s /original -d /backup --comparison binary
syncnorris compare -s /original -d /backup --comparison namesize
syncnorris compare -s /original -d /backup --comparison timestamp

# Display differences in JSON format
syncnorris compare -s /original -d /backup --diff-format json

# Save differences to a file instead of screen
syncnorris compare -s /original -d /backup --diff-report differences.txt

# The report includes:
# - Files with copy/update errors
# - Files only in source (not yet copied)
# - Files only in destination (only with --delete flag)
# - Files that would be deleted (with --delete flag)
# - Files with hash/content differences
# - Files skipped by exclude patterns
# - Detailed metadata (size, modification time, hash)

Exclude Files

# Exclude log files
syncnorris sync -s /src -d /dst --exclude "*.log"

# Exclude multiple patterns
syncnorris sync -s /src -d /dst --exclude "*.log" --exclude ".git/**" --exclude "node_modules/**"

# Excluded files appear in report with "skipped" reason

Bandwidth Limiting

# Limit transfer speed to 10 MiB/s
syncnorris sync -s /src -d /dst --bandwidth 10M

# Limit to 500 KiB/s (useful for slow networks)
syncnorris sync -s /src -d /dst -b 500K

# Limit to 1 GiB/s
syncnorris sync -s /src -d /dst -b 1G

# Bandwidth is applied to both file copying and hash comparison

JSON Output

# Get JSON output for automation
syncnorris sync -s /src -d /dst --output json

# Combine with diff-report for structured logging
syncnorris sync -s /src -d /dst --output json --diff-report sync.json --diff-format json

Generate Differences Report for Sync

# Sync normally doesn't show differences report
syncnorris sync -s /src -d /dst

# Save differences report to a file after sync
syncnorris sync -s /src -d /dst --diff-report sync_differences.txt

# Generate JSON report for automation
syncnorris sync -s /src -d /dst \
  --diff-report sync_report.json \
  --diff-format json

Maximum Performance

# Use more parallel workers for I/O-bound operations
syncnorris sync \
  -s /source \
  -d /dest \
  --parallel 16 \
  --comparison namesize

Fast Hash Verification

# Use MD5 for faster hash-based comparison (less secure than SHA-256)
syncnorris sync \
  -s /media/photos \
  -d /backup/photos \
  --comparison md5

Debugging File Differences

# Use binary comparison to find exact byte offset where files differ
syncnorris sync \
  -s /original \
  -d /modified \
  --comparison binary \
  --dry-run

Logging

# Enable file logging
syncnorris sync -s /src -d /dst --log-file /var/log/syncnorris.log

# Use JSON format for structured logging
syncnorris sync -s /src -d /dst --log-file sync.log --log-format json

# Enable debug-level logging for troubleshooting (traces every file operation)
syncnorris sync -s /src -d /dst --log-file debug.log --log-level debug

# Combine logging with other options
syncnorris sync -s /src -d /dst \
  --log-file /var/log/syncnorris.log \
  --log-format json \
  --log-level info

Debug log output example (text format):

2025-11-29T10:30:45Z [DEBUG] Processing file path=document.txt size=1024 worker=0 dest_exists=true
2025-11-29T10:30:45Z [DEBUG] File synchronized (identical) path=document.txt size=1024 duration=1.2ms
2025-11-29T10:30:45Z [DEBUG] Copying file (new) path=newfile.txt size=2048 dry_run=false
2025-11-29T10:30:45Z [DEBUG] File copied successfully path=newfile.txt size=2048 duration=5.3ms
2025-11-29T10:30:45Z [DEBUG] Updating file (content differs) path=modified.txt size=512 dry_run=false
2025-11-29T10:30:45Z [DEBUG] File updated successfully path=modified.txt size=512 duration=3.1ms

Performance Tips

First sync: Use --comparison hash (default) for cryptographic verification
Re-sync: Use --comparison namesize for 10-40x speedup on unchanged files
Fast hash: Use --comparison md5 for slightly faster hashing (less secure than SHA-256)
Debugging: Use --comparison binary for byte-by-byte verification with exact offset reporting
Large files: Hash comparison (SHA-256/MD5) automatically uses partial hashing (≥1MB)
Network storage: Mount shares locally rather than waiting for native SMB/NFS support
Worker count: Default is 5; increase for fast I/O or decrease for slow disks
Progress overhead: Already optimized (93% reduction), no tuning needed

Project Structure

syncnorris/
├── cmd/syncnorris/           # Main CLI entry point
├── pkg/                      # Public packages
│   ├── storage/              # Storage backends (local filesystem)
│   ├── compare/              # Comparison algorithms (hash, composite)
│   ├── sync/                 # Sync engine and worker pools
│   ├── output/               # Output formatters (human, progress)
│   ├── config/               # Configuration management
│   └── models/               # Data models and types
├── internal/                 # Private packages
│   └── cli/                  # CLI commands and validation
├── docs/                     # Optimization documentation
├── specs/                    # Feature specifications
├── scripts/                  # Build and test scripts
└── tests/                    # Test files

Development

Prerequisites

Go 1.21+ (uses sync/atomic and other modern features)
Make (optional but recommended)

Building

# Install dependencies
go mod download

# Build for current platform
make build

# Run tests
make test

# Cross-compile for all platforms
make build-all

Running Tests

# Unit tests
go test ./...

# With coverage
go test -cover ./...

# Performance benchmarks
go test -bench=. ./pkg/compare/

Documentation

IMPLEMENTATION_STATUS.md - Detailed feature status
CHANGELOG.md - Version history and optimization details
docs/ATOMIC_COUNTERS_OPTIMIZATION.md - Lock-free statistics
docs/PARALLEL_HASH_OPTIMIZATION.md - Concurrent hashing
docs/PARTIAL_HASH_OPTIMIZATION.md - Quick rejection strategy
docs/THROTTLE_OPTIMIZATION.md - Callback optimization

Known Limitations

Bidirectional sync is EXPERIMENTAL - functional but not production-ready
Network storage requires mounting (no native SMB/NFS support planned for post-v1.0)
Interrupted operations cannot be resumed (checkpointing planned for post-v1.0)

See IMPLEMENTATION_STATUS.md for complete list.

Roadmap

v0.6.0: Logging infrastructure ✅
v0.7.0+: Bisync stabilization
v1.0.0: Promote bidirectional sync to production-ready
Post-v1.0: Resume functionality, network backends (SMB/NFS)

Contributing

Contributions are welcome! Priority areas:

Testing and feedback on bidirectional sync
Documentation improvements
Bug reports and feature requests

License

This project is licensed under the MIT License - see the LICENSE file for details.

Third-Party Licenses

This project uses several open-source libraries. See THIRD_PARTY_LICENSES.md for detailed license information about dependencies.

Credits

Built with performance in mind, leveraging Go's excellent concurrency primitives and modern optimization techniques.

Directories ¶

Path	Synopsis
cmd
syncnorris command
internal
cli
platform
pkg
compare
config
logging
models
output
ratelimit
storage
sync

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL