Commit Graph

513 Commits (ba9d8d7920bcfb6e02ec07ca4a50deca2df0bf5e)

Author SHA1 Message Date
Tienson Qin 6df866a2d1 fix: handle more whiteboard properties 2024-04-08 22:56:14 +08:00
Tienson Qin 2940e86d74 fix: save whiteboard 2024-04-08 22:54:56 +08:00
charlie 618f3bd107 Merge branch 'master' into feat/db 2024-04-08 19:00:51 +08:00
Tienson Qin cd38e0b9f9 fix: calculate :block/path-refs when adding new graph or re-index 2024-04-08 17:24:51 +08:00
Tienson Qin 210be5aaca fix: creating redundant pages when editing a block 2024-04-08 01:25:48 +08:00
Tienson Qin a324d845b2 fix: don't parse refs from quoted string 2024-04-07 21:35:14 +08:00
Tienson Qin b5f0f58efc fix: missing refs for property text values when creating a page 2024-04-07 21:35:14 +08:00
Tienson Qin 0b49d09fb1 fix: db schema validate 2024-04-07 17:26:58 +08:00
Tienson Qin f52993ec6d fix: tests and lint warnings 2024-04-07 14:27:05 +08:00
Tienson Qin 6eb6e60b55 Bring back namespaces/page merge/nested pages 2024-04-07 13:48:55 +08:00
Tienson Qin 0bc8d1c3b1 Merge branch 'refactor/db-properties-schema' into refactor/db-remove-block-name-unique 2024-04-05 15:16:13 +08:00
Tienson Qin ac31842cec Revert "Remove :block/macros"
This reverts commit 21d550de12.
2024-04-05 15:12:56 +08:00
Tienson Qin ea98a1623e Merge branch 'refactor/db-properties-schema' into refactor/db-remove-block-name-unique 2024-04-05 09:56:39 +08:00
Gabriel Horner 7c1f52d2a4 chore: add format to build-new-page
every use of it was adding it manually
2024-04-04 09:25:20 -04:00
Tienson Qin 8054150eef fix: unit tests 2024-04-04 15:57:49 +08:00
Tienson Qin 6167abe160 fix: extract page from whiteboard edn 2024-04-04 15:57:49 +08:00
Tienson Qin 1f32893547 fix most issues on whiteboards 2024-04-04 15:57:48 +08:00
Tienson Qin 9df8918f94 Remove more :block/name lookup ref usage 2024-04-04 15:57:48 +08:00
Tienson Qin 60d4fca0ba wip: remove :block/name uniqueness for db based graphs
This PR also remove supports for:
1. merge pages when renaming a page to existing page
2. namespaces such as a/b/c
3. nested page such as [[a [[nested page]]]]

Pages merge might be added back depends on RTC, but it should be
decoupled from renaming, otherwise it's too complex.

Namespaces and nested pages have been contributed some critical bugs
that lead data-loss, they're so complex together with page alias,
it's just impossible to have a good test coverage and ensure the app
is stable, especially when page rename and RTC.
2024-04-04 15:57:48 +08:00
Tienson Qin 7084b53dad Merge branch 'feat/db' into refactor/db-properties-schema 2024-04-03 20:29:11 +08:00
Gabriel Horner 12c667fabb Update nbb-logseq to latest datascript 2024-04-02 09:36:28 -04:00
Gabriel Horner 93846fe62f enhance: add import support for priority, scheduled and deadline
fixes LOG-3030
2024-04-01 12:31:39 -04:00
Gabriel Horner 745fcdb7a3 enhance: import tasks
Part of LOG-3030
2024-04-01 10:48:25 -04:00
Tienson Qin 1987c455e7 Merge branch 'feat/db' into refactor/db-properties-schema 2024-04-01 15:47:43 +08:00
Tienson Qin d48e45eaa7 enhance: property db ident doesn't allow # 2024-04-01 14:53:25 +08:00
Tienson Qin 02643a1561 fix: tests 2024-04-01 14:25:39 +08:00
Tienson Qin 21d550de12 Remove :block/macros 2024-04-01 14:22:15 +08:00
Gabriel Horner 1d014c1326 fix graph-parser lint 2024-03-29 17:09:30 -04:00
Tienson Qin 441dda79c8 fix: typo 2024-03-30 05:02:01 +08:00
Gabriel Horner de89e65129 chore: remove unused fns and move file graph fn 2024-03-29 16:32:23 -04:00
Gabriel Horner f728b7b9f6 fix: multiple bugs with built-in pages, fixes LOG-3046
- file graph wasn't creating built in pages
- 3 places where db graphs were incorrectly using file graph built in
  page names
- file graphs creating default pages with incorrect format
2024-03-29 16:21:19 -04:00
Gabriel Horner e08e52acd3 enhance: db graph importer supports optional parent class property 2024-03-29 10:37:09 -04:00
Tienson Qin 958507262e Merge branch 'feat/db' into refactor/db-properties-schema 2024-03-29 15:51:28 +08:00
Gabriel Horner 2e580df0da chore: rename exporter built-in property
use dashes like other built-ins
2024-03-28 14:41:37 -04:00
Gabriel Horner e0c6cc1514 fix: deprecate :macros property for db graphs
It's not used much and would require building a new UI since
we don't have one yet for a :map type property
2024-03-28 09:13:39 -04:00
Tienson Qin 41e2944720 Replace more property name usage with db/ident 2024-03-28 20:41:12 +08:00
Gabriel Horner eecb510ccb enhance: built-in properties are indexed by :db/ident
built-in-properties are identified by their :db/ident instead
of :block/name. This removes reliance on :block/name and allows
us to consistently use namespaces when identifying built-in properties

:visible was renamed to :public? and is now included in :block/schema to
allow for querying. Also added a couple more new graph tests and tweaked
CI for db dep
2024-03-27 15:03:18 -04:00
Gabriel Horner 9b8540921e enhance: more property helper fns uses db-ident
instead of name
2024-03-25 16:34:51 -04:00
Tienson Qin eae35d60e4 fix: rebuild block refs & path-refs after downloading a remote graph
Fixes LOG-3115
2024-03-21 16:45:00 +08:00
Gabriel Horner 67a7a0a3e9 enhance: Add default namespace of :logseq.property
for db/ident of built-in-properties. Namespacing block identities will
be important for future features. Adding namespacing also fixed some
hidden coupling between identity and name. Also converted a couple more
related :block/name queries to use :db/ident to slowly get us closer to
named blocks
2024-03-15 18:37:15 -04:00
Gabriel Horner 345fd96e2a enhance: full file graph import works from CLI
Also cleaned up naming, added docstrings and fixed a bug that
failed import hard when user didn't input any tag-classes or
property-classes
2024-03-13 16:23:49 -04:00
Gabriel Horner 2f599919fb fix: favorites not importing
Also made underlying fns reusable from UI and nbb compatible.
Fixes LOG-3031
2024-03-12 17:57:47 -04:00
Gabriel Horner 96b35c0252 Update nbb-logseq to stub in cljs-bean.transit 2024-03-11 15:19:50 -04:00
Andelf 329818e4c4 fix(db): css assets loading
- move make-asset-url to assets-handler
- handle assets for db-based graph in add-style-if-exists!
2024-03-11 18:38:48 +08:00
Gabriel Horner c6e358c983 fix: outliner test failing b/c not nbb compatible
bumped to latest nbb-logseq which is
2024-03-06 11:06:18 -05:00
Gabriel Horner 09816b6427 enhance: assets can be imported on CLI
Also add individual asset error handling, like is done for doc files
2024-03-05 17:05:18 -05:00
Gabriel Horner 9669755cab enhance: detect and add properties to imported classes
fixes LOG-3069
2024-03-04 23:05:06 -05:00
Gabriel Horner b34325be77 fix: two bugs with updating tags during import
fixes LOG-3073
2024-03-01 15:51:19 -05:00
Gabriel Horner 487c86ac83 enhance: import float values with type :number
They were imported as :default b/c graph-parser didn't handle them.
Part of LOG-2985
2024-03-01 12:07:45 -05:00
Gabriel Horner 0b0299b28f fix: more import fixes after testing more graphs
- collapsed:: false was failing an import
- ignore more attributes for existing pages
- also fixed import script not handling macros and absolute paths
  correctly
2024-02-29 15:06:49 -05:00
Gabriel Horner 49cc17dea5 fix: update property values for any type to :default change
that occur within same file. Also made importing behavior reproducible
across runs and made import script script less verbose by default.
Part of LOG-2985
2024-02-29 13:01:09 -05:00
Gabriel Horner 31b344d3c1 enhance: ignore blank property values from templates
to improves property type detection. Also fixed a bug in
upstream-properties detection and added more docs + better names related
to it. Part of LOG-2985
2024-02-29 11:18:18 -05:00
Gabriel Horner b32eb7e9be enhance: automatically handle properties that change to :default
docs graph :platforms property now imports correctly rather than
ignored 30 times. Also allow import script to import multiple files
for testing purposes. Part of LOG-2985
2024-02-28 17:16:09 -05:00
Gabriel Horner 9569bed720 fix: property value with mixed refs not imported
correctly. Also fixed lints
2024-02-28 11:08:53 -05:00
Gabriel Horner 44d60808c1 fix: correctly detect property type
for property values containing text and refs.
Also allow import script to only import a file for testing.
Part of LOG-2985
2024-02-27 16:27:26 -05:00
Gabriel Horner 13439ed699 enhance: port two more import steps to be script compatible
also fix save-file! not waiting on async transact before post-save
2024-02-27 12:07:25 -05:00
Gabriel Horner 3e3e9a303b fix graph-parser lints 2024-02-26 16:30:12 -05:00
Gabriel Horner e3cc4bad36 enhance: speed up import of doc files and make them scriptable
Refactored the largest part of the import to used promesa which made it
nbb compatible and made large imports like docs 10-25% faster.  Also
updated db-import script to use this work
2024-02-26 16:20:03 -05:00
Gabriel Horner d8f41f22c2 fix: class page schemas have a new optional attribute 2024-02-26 11:11:01 -05:00
Tienson Qin b1aae957e0 fix: esc doesn't save tags 2024-02-24 16:21:03 +08:00
Gabriel Horner 02ba0b3c87 enhance: configurable logging and user notification during import
Don't need to sprinkle prn's anymore or hide errors that shouldn't
be happening while developing
2024-02-24 00:55:14 -05:00
Gabriel Horner a61ee21e25 enhance: add basic script to test db graph import
also tweaked query script to work with db graphs by relative path
2024-02-24 00:42:46 -05:00
Gabriel Horner 356657d36b fix: adding uuids on imported classes causes multiple bugs
Caused invalid blocks as properties that were on the same block
as new class had invalid uuids. Also caused content to have
incorrect mangled uuids. Part of LOG-2985
2024-02-23 15:23:47 -05:00
Gabriel Horner eb994662a6 enhance: user can choose properties' values to import as classes
The user specified properties are then no longer needed since the
concept is built-in. Part of LOG-2985
2024-02-23 12:42:16 -05:00
Gabriel Horner dad848fa73 enhance: improve property type detection
with macros. This improves :url property detection in docs
graph and other similar graphs. part of LOG-2985
2024-02-22 16:00:03 -05:00
Gabriel Horner 531a8ef533 enhance: import :filters and helpfully ignore :icon
Better to ignore :icon and explain to user than import them as broken with no explanation.
part of LOG-2985
2024-02-22 10:12:09 -05:00
Gabriel Horner 37135fd242 enhance: import tags for existing pages - LOG-2985
also fixed namespaces and aliases missing from existing pages
2024-02-21 17:16:21 -05:00
Gabriel Horner a010298fc6 enhance: handle invalid property value changes
Ignore invalid property values and notify user, rather than
import them as broken. Part of LOG-2985
2024-02-21 15:30:27 -05:00
Gabriel Horner 95a43e24a3 clean up property importing - part of LOG-2985
- infer block property values by properties, not refs
- remove unnecessary property-changes state
- centralize property-change logging and handling
2024-02-21 11:45:40 -05:00
Gabriel Horner 9591c51997 enhance: import properties of existing pages
including property pages. part of LOG-2985
2024-02-20 23:02:44 -05:00
Gabriel Horner ddefb677f2 enhance: import most page properties - part of LOG-2985
Also fix validation and bugs with built-in props. Add additional info
on parent-left conflicts as I encountered a couple while testing
pre-block removal
2024-02-20 14:50:49 -05:00
Gabriel Horner 4f3c4c9e03 fix: handle a property :default to :page change
Part of LOG-2985. :page property values would just be references so
use properties-text-values over `str`. Also added import timing
2024-02-19 11:34:42 -05:00
Gabriel Horner 7734bc8791 enhance: import can detect property type changes
and handle auto-converting a value changing from :default to any prop type
2024-02-17 00:24:50 -05:00
Gabriel Horner 04e5c79ee0 fix: invalid property values by unsupported template blocks
Correctly ignore template properties until it's supported
2024-02-16 12:36:27 -05:00
Gabriel Horner 3bf7ee5645 enhance: import support for :page and infer :date properties 2024-02-16 11:36:06 -05:00
Gabriel Horner 0bba0a2ca9 enhance: import built-in logseq.* props - part of LOG-2977 2024-02-15 15:01:01 -05:00
Gabriel Horner 18d3c87016 enhance: support built-in property imports. part of LOG-2985
:query-properties and :query-sort-by need translating. Don't
override block/schema of built-in properties. Also display
property closed values in block data cmd
2024-02-15 12:27:36 -05:00
Gabriel Horner 3b745ac035 enhance: first pass at importing block properties
part of LOG-2985. Still somethings failing for docs graph
2024-02-14 17:16:36 -05:00
Gabriel Horner 2ff0cc1702 enhance: import whiteboards
Fixes LOG-2981. Also start setting up block properties for LOG-2985
2024-02-13 15:41:56 -05:00
Gabriel Horner 48b974d64e enhance: all page refs in imported content are special ids
close LOG-3055. Also fix graph-parser errors being ignored
2024-02-12 22:02:54 -05:00
Gabriel Horner 6dc1272bf2 fix: lint and remove unused ast in file graph exporter
Also remove unused options on d/transact!
2024-02-12 15:52:52 -05:00
Gabriel Horner 06ad57efff fix: page tags causing invalid properties on larger graphs
Tags that had their own pages were resulting in invalid properties. This
is because we were re-transacting existing pages as new pages
with new uuids which resulted in invalid properties. Fixes LOG-2984.
Also moved graph-parser export (import from db perspective) to its own
ns
2024-02-12 12:33:13 -05:00
Gabriel Horner 26bc6f0d1d chore: mv logseq.graph-parser.util.db to db dep
This ns is more appropriate in db and was only in graph-parser because
the date-time-util lib wasn't accessible to db until #10933 landed.
graph-parser dep is specific to file graphs and as much as possible
anything unrelated to this should not be in this dep
2024-02-09 16:16:33 -05:00
Tienson Qin 62b90b7ae7
Merge branch 'feat/db' into perf/lazy-load-data 2024-02-09 09:28:36 +08:00
Gabriel Horner dab6b5aff4 enhance: add support for importing page and block tags
Allows users to specify which tags are tag classes and
the rest are just normal pages. Fixes LOG-2984
2024-02-08 16:49:27 -05:00
Tienson Qin 6f90958e4b Merge branch 'feat/db' into perf/lazy-load-data 2024-02-03 12:52:35 +08:00
Gabriel Horner ca752834c1 enhance: basic block tags for import - part of LOG-2984 2024-02-02 17:39:08 -05:00
Gabriel Horner b6a952fd7d fix: assets not copied for import
Caused by a commit yesterday. Also removed unused
import-file-to-db-graph options
2024-02-02 16:16:18 -05:00
Gabriel Horner 5b4bfd9089 enhance: file graph import ignores hidden files 2024-01-31 15:01:39 -05:00
Tienson Qin c1a15c0708 Merge branch 'feat/db' into perf/lazy-load-data 2024-02-01 03:58:04 +08:00
Gabriel Horner bf56cd3252 chore: move legacy fns to legacy ns
title property is for file graphs and should be kept with its ns
to avoid confusion
2024-01-31 11:11:14 -05:00
Tienson Qin 99b6d543f0 fix: more async queries and export to roam 2024-01-30 20:03:06 +08:00
Tienson Qin 3addb5905b perf: load partial data when loading a graph
Instead of all the datoms
2024-01-29 13:17:37 +08:00
Gabriel Horner 77bc1d5d6b chore: update better-sqlite3 and mldoc in deps to match app 2024-01-23 11:36:32 -05:00
Gabriel Horner 689cf34cc3 fix: lints and a test
nbb test was failing b/c outliner had graph-parser as a dependency
but we hadn't manually added the npm dependency
2024-01-23 09:38:16 -05:00
Gabriel Horner f573b8e821 fix: file graph import not persisting to disk 2024-01-17 12:57:16 -05:00
Gabriel Horner 0168ef0082 fix: 2 bugs with file import
These were silently failing and causing a number of pages to
not import
2024-01-16 17:01:23 -05:00
Tienson Qin cea105cc34 fix: wrong ns import 2024-01-16 16:01:18 +08:00
Tienson Qin a619850998 Merge branch 'feat/db' into refactor/pipeline-worker 2024-01-16 15:42:18 +08:00
Gabriel Horner 34222b3436 chore: bump nbb-logseq that points to upstream nbb 2024-01-15 18:07:49 -05:00