first commit

This commit is contained in:
2025-11-14 21:37:55 +01:00
parent 86a5de420c
commit 2464da9f7d
14 changed files with 4100 additions and 0 deletions

1
.gitignore vendored Normal file
View File

@@ -0,0 +1 @@
/target/

View File

@@ -29,6 +29,16 @@ The local directory structure must be: `<semester>/<course>/<studip-folders>/<fi
- `GET /users/{user_id}/courses` to list enrolled courses. [web:85]
- Course-specific routes for folders and documents/file-refs, using the documented JSON:API routes for Stud.IP (e.g. `/courses/{course_id}/documents`). [web:88][web:93]
### Field notes (2025-02-16)
- `/users/me` returns the canonical user ID (`cbcee42edfea…`), full profile attributes, and relationship URLs (courses, folders, file-refs, etc.). Cache the `id` immediately so later runs can skip this discovery call unless credentials change.
- `/users/{id}/courses` is paginated via `meta.page { offset, limit, total }` and `links.first/last` (e.g. `/jsonapi.php/v1/users/.../courses?page[offset]=0&page[limit]=30`). Default limit is 30; loop by bumping `offset` until `offset >= total`. Each course provides `start-semester`/`end-semester` relationships to semester IDs, course numbers, and titles.
- `/semesters/{id}` exposes only human strings like `"WiSe 2024/25"` plus ISO start/end timestamps—no canonical short keys. Derive keys such as `ws2425` from the title or `start` year and cache the mapping `semester_id → key` in `state.toml`.
- `/courses/{id}/folders` lists folder nodes with attributes (`folder-type`, `is-empty`, mkdate/chdate) and nested relationships: follow `/folders/{folder_id}/folders` recursively for subfolders, because `meta.count` only reports a child count.
- `/folders/{id}/file-refs` is the primary listing for downloadable files. Each `file-ref` has attributes (`name`, `filesize`, `mkdate`, `chdate`, MIME, `is-downloadable`), relationships back to the parent folder/course, and a `meta.download-url` like `/sendfile.php?...`. Prepend the configured base URL before downloading.
- `/files/{id}` only repeats size/timestamp data and links back to `file-refs`; it does **not** expose checksums. Track change detection via `(file-ref id, filesize, chdate)` and/or compute local hashes.
- File/folder listings share the same JSON:API pagination scheme. Always honor the `meta.page` counts and `links.first/last/next` to avoid missing entries in large folders.
## Configuration (TOML, including paths)
All configuration and state in this project must use **TOML**. [web:131]

2264
Cargo.lock generated Normal file

File diff suppressed because it is too large Load Diff

24
Cargo.toml Normal file
View File

@@ -0,0 +1,24 @@
[package]
name = "studip-sync"
version = "0.1.0"
edition = "2024"
[dependencies]
anyhow = "1.0"
base64 = "0.22"
clap = { version = "4.5", features = ["derive"] }
directories = "5.0"
reqwest = { version = "0.12", default-features = false, features = ["json", "stream", "gzip", "brotli", "deflate", "rustls-tls"] }
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"
thiserror = "1.0"
tokio = { version = "1.37", features = ["macros", "rt-multi-thread", "fs", "io-util"] }
toml = "0.8"
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter", "fmt", "json", "time"] }
url = "2.4"
atty = "0.2"
rpassword = "7.3"
walkdir = "2.5"
time = "0.3"
sha2 = "0.10"

20
README.md Normal file
View File

@@ -0,0 +1,20 @@
# studip-sync
Command-line tool written in Rust (edition 2024) to sync files from the Stud.IP JSON:API to a local filesystem tree.
The repository contains the cargo project (with CLI/config/state scaffolding) plus an offline copy of the JSON:API documentation (in `jsonapi/`).
## Current status
- `cargo` binary crate scaffolded with name `studip-sync`, pinned to Rust edition 2024.
- CLI implemented with `auth`, `sync`, and `list-courses` subcommands plus logging/verbosity flags.
- Config/state loaders wired up with XDG path resolution, multi-profile support, and JSON/quiet/debug logging modes.
- `studip-sync auth` prompts for credentials (or reads `--username/--password` / `STUDIP_SYNC_USERNAME|PASSWORD`) and stores the base64 Basic auth token in the active profile.
- `studip-sync list-courses` now talks to the Stud.IP JSON:API, caches user/semester/course metadata, and prints a table of enrolled courses (with pagination + semester-key inference).
- `studip-sync sync` walks courses → folders → file refs via the JSON:API, downloads missing or changed files (streamed to disk), and supports `--dry-run` / `--prune` cleanup.
- Ready for further implementation of Stud.IP HTTP client, sync logic, and actual command behaviors.
## Next steps
1. Add configurable download concurrency plus richer progress/logging (per-course summaries, ETA) while keeping memory usage low.
2. Implement smarter state usage (incremental `filter[since]` queries, resume checkpoints) and expand pruning to detect/cleanup orphaned state entries.
3. Add tests and ensure `cargo fmt` + `cargo clippy --all-targets --all-features -- -D warnings` + `cargo test` pass (and wire into CI if applicable).

1032
src/cli.rs Normal file

File diff suppressed because it is too large Load Diff

114
src/config.rs Normal file
View File

@@ -0,0 +1,114 @@
use crate::Result;
use anyhow::Context;
use serde::{Deserialize, Serialize};
use std::{
collections::HashMap,
fs::{self, OpenOptions},
io::Write,
path::{Path, PathBuf},
};
const DEFAULT_PROFILE_NAME: &str = "default";
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ConfigProfile {
#[serde(default)]
pub username: Option<String>,
#[serde(default = "ConfigProfile::default_base_url")]
pub base_url: String,
#[serde(default = "ConfigProfile::default_jsonapi_path")]
pub jsonapi_path: String,
#[serde(default)]
pub basic_auth_b64: Option<String>,
#[serde(default)]
pub download_root: Option<PathBuf>,
#[serde(default = "ConfigProfile::default_max_concurrent_downloads")]
pub max_concurrent_downloads: usize,
}
impl Default for ConfigProfile {
fn default() -> Self {
Self {
username: None,
base_url: Self::default_base_url(),
jsonapi_path: Self::default_jsonapi_path(),
basic_auth_b64: None,
download_root: None,
max_concurrent_downloads: Self::default_max_concurrent_downloads(),
}
}
}
impl ConfigProfile {
fn default_base_url() -> String {
"https://studip.uni-trier.de".to_string()
}
fn default_jsonapi_path() -> String {
"/jsonapi.php/v1".to_string()
}
fn default_max_concurrent_downloads() -> usize {
3
}
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ConfigFile {
#[serde(default = "default_profile_name")]
pub default_profile: String,
#[serde(default)]
pub profiles: HashMap<String, ConfigProfile>,
}
impl Default for ConfigFile {
fn default() -> Self {
let mut profiles = HashMap::new();
profiles.insert(DEFAULT_PROFILE_NAME.to_string(), ConfigProfile::default());
Self {
default_profile: DEFAULT_PROFILE_NAME.to_string(),
profiles,
}
}
}
fn default_profile_name() -> String {
DEFAULT_PROFILE_NAME.to_string()
}
impl ConfigFile {
pub fn load_or_default(path: &Path) -> Result<Self> {
match fs::read_to_string(path) {
Ok(contents) => {
let file: Self =
toml::from_str(&contents).context("Failed to parse config.toml")?;
Ok(file)
}
Err(err) if err.kind() == std::io::ErrorKind::NotFound => Ok(Self::default()),
Err(err) => Err(err.into()),
}
}
pub fn save(&self, path: &Path) -> Result<()> {
if let Some(parent) = path.parent() {
fs::create_dir_all(parent)?;
}
let contents = toml::to_string_pretty(self)?;
let mut file = OpenOptions::new()
.create(true)
.write(true)
.truncate(true)
.open(path)?;
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt;
let mut perms = file.metadata()?.permissions();
perms.set_mode(0o600);
file.set_permissions(perms)?;
}
file.write_all(contents.as_bytes())?;
Ok(())
}
}

9
src/lib.rs Normal file
View File

@@ -0,0 +1,9 @@
pub mod cli;
pub mod config;
pub mod logging;
pub mod paths;
pub mod semesters;
pub mod state;
pub mod studip_client;
pub type Result<T> = anyhow::Result<T>;

61
src/logging.rs Normal file
View File

@@ -0,0 +1,61 @@
use crate::Result;
use tracing::Level;
use tracing_subscriber::{EnvFilter, fmt, prelude::*};
#[derive(Clone, Debug, Default)]
pub struct LogConfig {
pub quiet: bool,
pub debug: bool,
pub json: bool,
pub verbosity: u8,
}
impl LogConfig {
pub fn level(&self) -> Level {
if self.quiet {
return Level::ERROR;
}
if self.debug {
return Level::DEBUG;
}
match self.verbosity {
0 => Level::INFO,
1 => Level::DEBUG,
_ => Level::TRACE,
}
}
}
pub fn init_logging(cfg: &LogConfig) -> Result<()> {
let level = cfg.level();
let env_filter =
EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new(level.to_string()));
let registry = tracing_subscriber::registry().with(env_filter);
let init_result = if cfg.json {
registry
.with(
fmt::layer()
.with_target(false)
.with_level(true)
.with_ansi(atty::is(atty::Stream::Stdout))
.with_timer(fmt::time::SystemTime)
.json(),
)
.try_init()
} else {
registry
.with(
fmt::layer()
.with_target(false)
.with_level(true)
.with_ansi(atty::is(atty::Stream::Stdout))
.with_timer(fmt::time::SystemTime),
)
.try_init()
};
init_result.map_err(|err| err.into())
}

8
src/main.rs Normal file
View File

@@ -0,0 +1,8 @@
use clap::Parser;
use studip_sync::{Result, cli::Cli};
#[tokio::main]
async fn main() -> Result<()> {
let cli = Cli::parse();
cli.run().await
}

58
src/paths.rs Normal file
View File

@@ -0,0 +1,58 @@
use crate::Result;
use anyhow::anyhow;
use directories::ProjectDirs;
use std::path::PathBuf;
#[derive(Debug, Clone, Default)]
pub struct PathOverrides {
pub config_dir: Option<PathBuf>,
pub data_dir: Option<PathBuf>,
}
#[derive(Debug, Clone)]
pub struct AppPaths {
pub config_dir: PathBuf,
pub data_dir: PathBuf,
}
impl AppPaths {
pub fn new(overrides: &PathOverrides) -> Result<Self> {
if let (Some(config_dir), Some(data_dir)) = (&overrides.config_dir, &overrides.data_dir) {
return Ok(Self {
config_dir: config_dir.clone(),
data_dir: data_dir.clone(),
});
}
let project_dirs = ProjectDirs::from("de", "UniTrier", "studip-sync")
.ok_or_else(|| anyhow!("Failed to determine XDG directories"))?;
let config_dir = overrides
.config_dir
.clone()
.unwrap_or_else(|| project_dirs.config_dir().to_path_buf());
let data_dir = overrides
.data_dir
.clone()
.unwrap_or_else(|| project_dirs.data_dir().to_path_buf());
Ok(Self {
config_dir,
data_dir,
})
}
pub fn ensure_dirs(&self) -> Result<()> {
std::fs::create_dir_all(&self.config_dir)?;
std::fs::create_dir_all(&self.data_dir)?;
Ok(())
}
pub fn config_file(&self) -> PathBuf {
self.config_dir.join("config.toml")
}
pub fn state_file(&self) -> PathBuf {
self.data_dir.join("state.toml")
}
}

107
src/semesters.rs Normal file
View File

@@ -0,0 +1,107 @@
pub fn infer_key(title: &str) -> String {
use SemesterSeason::*;
let season = detect_season(title);
let numbers = extract_numbers(title);
match season {
Winter => {
let first = numbers
.get(0)
.map(|value| last_two_digits(value))
.unwrap_or_else(|| "00".into());
let second = numbers
.get(1)
.map(|value| last_two_digits(value))
.unwrap_or_else(|| {
// winter spans two years; if only one provided, assume +1
numbers
.get(0)
.map(|n| increment_two_digits(n))
.unwrap_or_else(|| "00".into())
});
format!("ws{first}{second}")
}
Summer => {
let year = numbers
.get(0)
.map(|value| last_two_digits(value))
.unwrap_or_else(|| "00".into());
format!("ss{year}")
}
Unknown => fallback_key(title),
}
}
#[derive(Copy, Clone, Debug)]
enum SemesterSeason {
Winter,
Summer,
Unknown,
}
fn detect_season(title: &str) -> SemesterSeason {
let lower = title.to_ascii_lowercase();
if lower.contains("wise")
|| lower.contains("winter")
|| lower.contains("ws ")
|| lower.starts_with("ws")
{
SemesterSeason::Winter
} else if lower.contains("sose")
|| lower.contains("sommer")
|| lower.contains("summer")
|| lower.contains("ss ")
|| lower.starts_with("ss")
{
SemesterSeason::Summer
} else {
SemesterSeason::Unknown
}
}
fn extract_numbers(title: &str) -> Vec<String> {
let mut numbers = Vec::new();
let mut current = String::new();
for ch in title.chars() {
if ch.is_ascii_digit() {
current.push(ch);
} else if !current.is_empty() {
numbers.push(current.clone());
current.clear();
}
}
if !current.is_empty() {
numbers.push(current);
}
numbers
}
fn last_two_digits(value: &str) -> String {
if value.len() <= 2 {
format!("{:0>2}", value)
} else {
value[value.len() - 2..].to_string()
}
}
fn increment_two_digits(value: &str) -> String {
let last_two = last_two_digits(value);
let parsed = last_two.parse::<u8>().unwrap_or(0);
format!("{:02}", (parsed + 1) % 100)
}
fn fallback_key(title: &str) -> String {
let mut key = String::from("sem-");
for ch in title.chars() {
if ch.is_ascii_alphanumeric() {
key.push(ch.to_ascii_lowercase());
} else if ch.is_whitespace() && !key.ends_with('-') {
key.push('-');
}
if key.len() >= 32 {
break;
}
}
key.trim_end_matches('-').to_string()
}

85
src/state.rs Normal file
View File

@@ -0,0 +1,85 @@
use crate::Result;
use serde::{Deserialize, Serialize};
use std::{
collections::HashMap,
fs,
path::{Path, PathBuf},
};
#[derive(Debug, Default, Serialize, Deserialize)]
pub struct StateFile {
#[serde(default)]
pub profiles: HashMap<String, ProfileState>,
}
#[derive(Debug, Default, Serialize, Deserialize)]
pub struct ProfileState {
#[serde(default)]
pub user_id: Option<String>,
#[serde(default)]
pub semesters: HashMap<String, SemesterState>,
#[serde(default)]
pub courses: HashMap<String, CourseState>,
#[serde(default)]
pub files: HashMap<String, FileState>,
}
#[derive(Debug, Default, Serialize, Deserialize, Clone)]
pub struct SemesterState {
pub id: String,
pub title: String,
pub key: String,
}
#[derive(Debug, Default, Serialize, Deserialize, Clone)]
pub struct CourseState {
pub id: String,
pub name: String,
pub semester_key: String,
#[serde(default)]
pub last_sync: Option<String>,
}
#[derive(Debug, Default, Serialize, Deserialize, Clone)]
pub struct FileState {
pub id: String,
#[serde(default)]
pub size: Option<u64>,
#[serde(default)]
pub checksum: Option<String>,
#[serde(default)]
pub last_downloaded: Option<String>,
#[serde(default)]
pub remote_modified: Option<String>,
#[serde(default)]
pub path_hint: Option<PathBuf>,
}
impl StateFile {
pub fn load_or_default(path: &Path) -> Result<Self> {
match fs::read_to_string(path) {
Ok(contents) => Ok(toml::from_str(&contents)?),
Err(err) if err.kind() == std::io::ErrorKind::NotFound => Ok(Self::default()),
Err(err) => Err(err.into()),
}
}
pub fn save(&self, path: &Path) -> Result<()> {
if let Some(parent) = path.parent() {
fs::create_dir_all(parent)?;
}
let contents = toml::to_string_pretty(self)?;
fs::write(path, contents)?;
Ok(())
}
pub fn profile_mut(&mut self, profile: &str) -> &mut ProfileState {
self.profiles
.entry(profile.to_string())
.or_insert_with(ProfileState::default)
}
pub fn profile(&self, profile: &str) -> Option<&ProfileState> {
self.profiles.get(profile)
}
}

307
src/studip_client.rs Normal file
View File

@@ -0,0 +1,307 @@
use crate::{Result, config::ConfigProfile};
use anyhow::{Context, anyhow, bail};
use reqwest::{
Client, Response,
header::{AUTHORIZATION, HeaderValue},
};
use serde::{Deserialize, de::DeserializeOwned};
use url::Url;
#[derive(Clone)]
pub struct StudipClient {
http: Client,
base: Url,
root: Url,
auth_header: HeaderValue,
}
impl StudipClient {
pub fn from_profile(profile: &ConfigProfile) -> Result<Self> {
let auth_b64 = profile.basic_auth_b64.as_ref().ok_or_else(|| {
anyhow!("No credentials configured for this profile. Run `studip-sync auth` first.")
})?;
let mut auth_header = HeaderValue::from_str(&format!("Basic {}", auth_b64))
.context("Invalid base64 credentials")?;
auth_header.set_sensitive(true);
let (base, root) = build_root_and_api_urls(profile)?;
let http = Client::builder()
.user_agent("studip-sync/0.1")
.build()
.context("Failed to build HTTP client")?;
Ok(Self {
http,
base,
root,
auth_header,
})
}
pub async fn current_user(&self) -> Result<UserResponse> {
let url = self.endpoint("users/me")?;
self.get_json(url).await
}
pub async fn semester(&self, semester_id: &str) -> Result<SemesterResponse> {
let url = self.endpoint(&format!("semesters/{semester_id}"))?;
self.get_json(url).await
}
pub async fn list_courses(&self, user_id: &str) -> Result<Vec<Course>> {
let path = format!("users/{user_id}/courses");
self.fetch_all_pages(&path).await
}
pub async fn list_course_folders(&self, course_id: &str) -> Result<Vec<Folder>> {
let path = format!("courses/{course_id}/folders");
self.fetch_all_pages(&path).await
}
pub async fn list_subfolders(&self, folder_id: &str) -> Result<Vec<Folder>> {
let path = format!("folders/{folder_id}/folders");
self.fetch_all_pages(&path).await
}
pub async fn list_file_refs(&self, folder_id: &str) -> Result<Vec<FileRef>> {
let path = format!("folders/{folder_id}/file-refs");
self.fetch_all_pages(&path).await
}
pub async fn download_file(&self, relative_path: &str) -> Result<Response> {
let url = self.download_endpoint(relative_path)?;
self.send_request(url).await
}
fn endpoint(&self, path: &str) -> Result<Url> {
let normalized = path.trim_start_matches('/');
self.root.join(normalized).map_err(Into::into)
}
async fn get_json<T>(&self, url: Url) -> Result<T>
where
T: DeserializeOwned,
{
self.send_request(url)
.await?
.json::<T>()
.await
.map_err(Into::into)
}
async fn fetch_all_pages<T>(&self, path: &str) -> Result<Vec<T>>
where
T: DeserializeOwned,
{
let mut offset = 0usize;
let limit = 100usize;
let mut items = Vec::new();
loop {
let mut url = self.endpoint(path)?;
{
let mut pairs = url.query_pairs_mut();
pairs.append_pair("page[offset]", &offset.to_string());
pairs.append_pair("page[limit]", &limit.to_string());
}
let ListResponse { data, meta, .. } = self.get_json(url).await?;
let count = data.len();
items.extend(data);
let total = meta
.and_then(|meta| meta.page)
.map(|page| page.total)
.unwrap_or(offset + count);
if offset + limit >= total || count == 0 {
break;
}
offset += limit;
}
Ok(items)
}
fn download_endpoint(&self, path: &str) -> Result<Url> {
let normalized = path.trim_start_matches('/');
self.base.join(normalized).map_err(Into::into)
}
async fn send_request(&self, url: Url) -> Result<Response> {
let response = self
.http
.get(url.clone())
.header(AUTHORIZATION, self.auth_header.clone())
.send()
.await
.with_context(|| format!("GET {}", url))?;
if !response.status().is_success() {
let status = response.status();
let body = response.text().await.unwrap_or_default();
bail!("Stud.IP request failed ({status}) - {body}");
}
Ok(response)
}
}
fn build_root_and_api_urls(profile: &ConfigProfile) -> Result<(Url, Url)> {
let base = profile.base_url.trim_end_matches('/');
let jsonapi_path = profile.jsonapi_path.trim_start_matches('/');
let base_url = Url::parse(base).context("Invalid base_url")?;
let mut api_url = base_url.clone();
api_url.set_path(jsonapi_path);
if !api_url.path().ends_with('/') {
api_url.set_path(&format!("{}/", api_url.path().trim_end_matches('/')));
}
Ok((base_url, api_url))
}
#[derive(Debug, Deserialize)]
pub struct UserResponse {
pub data: UserData,
}
#[derive(Debug, Deserialize)]
pub struct UserData {
pub id: String,
pub attributes: UserAttributes,
}
#[derive(Debug, Deserialize)]
pub struct UserAttributes {
pub username: String,
#[serde(rename = "formatted-name")]
pub formatted_name: String,
}
#[derive(Debug, Deserialize)]
pub struct SemesterResponse {
pub data: SemesterData,
}
#[derive(Debug, Deserialize)]
pub struct SemesterData {
pub id: String,
pub attributes: SemesterAttributes,
}
#[derive(Debug, Deserialize)]
pub struct SemesterAttributes {
pub title: String,
}
#[derive(Debug, Deserialize)]
pub struct Course {
pub id: String,
pub attributes: CourseAttributes,
pub relationships: CourseRelationships,
}
#[derive(Debug, Deserialize)]
pub struct CourseAttributes {
#[serde(rename = "course-number")]
pub course_number: Option<String>,
pub title: String,
#[serde(default)]
pub subtitle: Option<String>,
}
#[derive(Debug, Deserialize)]
pub struct CourseRelationships {
#[serde(rename = "start-semester")]
pub start_semester: Option<Relationship>,
#[serde(rename = "end-semester")]
pub end_semester: Option<Relationship>,
}
impl CourseRelationships {
pub fn semester_id(&self) -> Option<&str> {
self.start_semester
.as_ref()
.and_then(|rel| rel.data.as_ref())
.map(|link| link.id.as_str())
}
}
#[derive(Debug, Deserialize)]
pub struct Relationship {
pub data: Option<ResourceIdentifier>,
}
#[derive(Debug, Deserialize)]
pub struct ResourceIdentifier {
#[serde(rename = "type")]
pub resource_type: String,
pub id: String,
}
#[derive(Debug, Deserialize)]
struct ListResponse<T> {
pub data: Vec<T>,
pub meta: Option<ListMeta>,
}
#[derive(Debug, Deserialize)]
struct ListMeta {
pub page: Option<PageMeta>,
}
#[derive(Debug, Deserialize)]
struct PageMeta {
#[allow(dead_code)]
pub offset: usize,
#[allow(dead_code)]
pub limit: usize,
pub total: usize,
}
#[derive(Debug, Deserialize, Clone)]
pub struct Folder {
pub id: String,
pub attributes: FolderAttributes,
}
#[derive(Debug, Deserialize, Clone)]
pub struct FolderAttributes {
#[serde(rename = "folder-type")]
pub folder_type: String,
pub name: String,
#[serde(rename = "is-readable")]
pub is_readable: bool,
#[serde(rename = "is-visible")]
pub is_visible: bool,
}
#[derive(Debug, Deserialize, Clone)]
pub struct FileRef {
pub id: String,
pub attributes: FileRefAttributes,
pub meta: Option<FileRefMeta>,
}
#[derive(Debug, Deserialize, Clone)]
pub struct FileRefAttributes {
pub name: String,
#[serde(rename = "mkdate")]
pub created: String,
#[serde(rename = "chdate")]
pub modified: String,
#[serde(rename = "filesize")]
#[serde(default)]
pub file_size: Option<u64>,
#[serde(rename = "mime-type")]
#[serde(default)]
pub mime_type: Option<String>,
}
#[derive(Debug, Deserialize, Clone)]
pub struct FileRefMeta {
#[serde(rename = "download-url")]
pub download_url: String,
}