Commit Graph

173 Commits (7c641925be316bf9d57f88399f30f9c11e20e3d4)

Author SHA1 Message Date
Tienson Qin b294896f8a fix: reuse page uuids when parsing files
This commit also removes `with-id?` because it's too confusing.

Fixes LOG-3143
2024-06-15 22:52:30 +08:00
Tienson Qin 4281274512 fix: graph parser tests 2024-06-15 21:08:25 +08:00
Gabriel Horner 9a8c91c0cc chore: mv logseq.db.sqlite.db into sqlite.cli
for more explicit name and purpose. Also updated outdated db readme
and removed unused fns in logseq.db.sqlite.db
2024-06-14 16:53:19 -04:00
Gabriel Horner 62701e1b6e fix: tests with outdated datom counts
I think caused by recent macros parsing or whiteboard changes
2024-06-05 17:06:16 -04:00
Gabriel Horner 602b236a4c fix: graph-parser lint 2024-05-02 10:23:20 -04:00
Tienson Qin 1ec4796eff refactor: replace :block/left with :block/order
Finally no need to worry about parent-left conflicts and broken chain.
With :block/order, we only need to re-compute new orders for siblings
with same order (it can happens if there're bugs in our code, or
updates from rtc), but it doesn't break UI.

Another huge potential benefit after discussing with Zhiyuan is:
Ee might be able to simplify both RTC and undo/redo, currently, we
need to handle each new op for both of them, with recently
refactorings like properties being db attributes, :block/order
is a string instead of a ref, we can handle most property value
conflicts using last-write-wins, and others (e.g. :block/parent,
property with :default type) specifically.

I haven't fixed the issues of using :block/left in RTC and undo/redo,
because we might change both soon.
2024-05-02 01:12:22 +08:00
Gabriel Horner 3681ac354b fix: move db-pipeline and datascript-report back to outliner dep
graph-parser is primarily a file graph dep that is maintained separately
from db graphs. We want to move as many namespaces out of it as
possible. Also moved differing count assertions to
their respective tests
2024-04-25 11:22:14 -04:00
Tienson Qin b55c064305 fix: move :block/path-res calculation to dep graph-parser 2024-04-25 16:50:04 +08:00
Gabriel Horner 878d69f342 fix: graph parser tests
extract and dependent namespaces require a db now and :block/path-refs
is no longer being handled in this dep
2024-04-24 16:55:24 -04:00
Tienson Qin 8d9e6da41e refactor: use :block/type for journal pages instead of :block/journal?
breaking change:
removes :block/journal attribute
2024-04-24 20:12:15 +08:00
Tienson Qin 9df8918f94 Remove more :block/name lookup ref usage 2024-04-04 15:57:48 +08:00
Gabriel Horner f728b7b9f6 fix: multiple bugs with built-in pages, fixes LOG-3046
- file graph wasn't creating built in pages
- 3 places where db graphs were incorrectly using file graph built in
  page names
- file graphs creating default pages with incorrect format
2024-03-29 16:21:19 -04:00
Gabriel Horner bf56cd3252 chore: move legacy fns to legacy ns
title property is for file graphs and should be kept with its ns
to avoid confusion
2024-01-31 11:11:14 -05:00
Gabriel Horner 951e7cc841 fix: ns-docstring and large-var deps lints
Also fixed db-and-file-graphs-separate lint and moved gp-property tests
that should be moved since their source code has moved from frontend
to gp-property
2024-01-12 17:03:09 -05:00
Gabriel Horner cdddce0120 fix deps db test and most lints
Also deleted duplicated gp-util.{page-ref,block-ref} files that were moved to common
and moved a related test to common dep
2024-01-11 14:35:04 -05:00
Tienson Qin 7258cf48da fix deps 2024-01-03 18:12:12 +08:00
Tienson Qin 9194a671fd Move outliner core to its own dep 2024-01-03 18:02:48 +08:00
Tienson Qin a6f83c2445 Move worker.util fns to common.util 2024-01-03 17:08:24 +08:00
Tienson Qin b7d9098e80 Move gp-util and gp-config to the common dep 2024-01-03 16:38:05 +08:00
Gabriel Horner ffa6aaae29 fix: remove unused db.sqlite related fns
- remove sqlite-db/close! as its not used and didn't make sense to be
  used in other electron ns
- remove sqlite-db/connections atom as its not needed
- remove read-graph as its not necessary
- rename cli.persist-graph as its primary purpose has changed
- update effected scripts
2023-12-13 12:06:24 -05:00
Gabriel Horner b5eabc80ee fix: db and graph-parser nbb tests to pass locally
Updated sqlite fns to use updated approach. Deleted some unused
fns from old approach
2023-12-11 13:51:47 -05:00
Gabriel Horner edaf9b286a chore: move frontend db namespaces to a dedicated parent ns
Moved these namespaces to logseq.db.frontend.* to make their purpose
explicit and make the namespaces easier to maintain
2023-10-16 17:24:51 -04:00
Gabriel Horner b1b830c872 fix tests I forgot to update with :block/format fix 2023-10-16 16:14:53 -04:00
Gabriel Horner e09a5d9988 fix: pages didn't consistenly have :block/format
Making attributes consistent reduce bugs in components and query layer (where
most wouldn't know to use get-else)
2023-10-16 12:58:11 -04:00
Gabriel Horner 82994cb79d fix: tests related to :block/type arity change 2023-09-25 12:30:21 -04:00
Tienson Qin 0e8653985d refactor: allow a block to have multiple types
For example, a whiteboard page can be used both as a property and a class.
2023-09-25 10:46:00 +08:00
Gabriel Horner 172ee0ae24 Provide sqlite.cli for reusable sqlite fns
Already had to do read-graph 3 times and will soon need to it outside
logseq so worth putting this behind an api ns
2023-08-15 18:08:44 -04:00
Gabriel Horner 40237b912f Remove nonexistent :block/unordered from test 2023-07-24 12:32:08 -04:00
Gabriel Horner 5f91e6cf0f Merge branch 'master' into feat/db 2023-07-24 12:21:19 -04:00
Gabriel Horner 2b90b0462a Remove explicit :block/unordered that should already be deleted 2023-07-24 10:26:02 -04:00
Gabriel Horner 48009adaf9 chore: reuse common helper for stringifying datoms 2023-07-20 18:37:22 -04:00
Gabriel Horner d7eeea28dd Add a test that ensures file and db graphs have expected datoms 2023-07-20 18:27:07 -04:00
Andelf a59ecb370d chore(deps): bump diff-merge to 0.1.0 2023-06-26 19:20:04 +08:00
Junyi Du bea081c5b0 test: more remove indent test 2023-06-16 00:42:30 +08:00
Tienson Qin bccf5888c7 add tests 2023-06-16 00:42:30 +08:00
Gabriel Horner 51a23934a4 Fix parse-graph not ignoring correctly for relative dir
parse-graph now returns absolute file paths
2023-05-19 21:49:58 -04:00
Gabriel Horner 47508ac2ac Add :hidden config support to graph-parser.cli
Also fix lints and tests from previous commits
2023-05-18 23:25:15 +08:00
Gabriel Horner 91beda09d5 Split out graph fns and reuse them b/n gp-cli and electron
Also added test
2023-05-18 23:25:15 +08:00
Gabriel Horner fe24b61beb Fix asset link creating invalid page for org mode 2023-05-15 11:55:59 +08:00
Gabriel Horner 7335a6f623 Fix org mode page refs - close #9367 2023-05-15 11:55:59 +08:00
Gabriel Horner b26d83d2fc Fix mailto and other links creating invalid pages
Fixes #9346 and fixes #5926.
Also fixes case where relative paths for unsupported paths were creating
pages per directory e.g. `deps/graph-parser/yarn.lock` created deps and
graph-parser pages
2023-05-11 13:56:25 +08:00
Junyi Du 5aba871ead deps: diff-merge
dev: graph parser IoC hook

test: use test db for diff-merge tests

fix: ci lint

dev: refactoring post block-parsing process

feat: diff-merge 2 way merge integration

fix: key namespace of uuid in fix-duplicated-id

fix: duplicated uuid ci
2023-04-17 09:41:39 -04:00
Gabriel Horner eac3c273ca Updates docs version for parse-graph integration tests 2023-04-14 09:35:55 -04:00
Gabriel Horner 5cce967192 Add nbb-test-runner to graph-parser
Replaces manual, hardcoded runner
2023-04-12 11:02:43 +08:00
Gabriel Horner 1c3ef2ccb5 Fix url? detection for raw paste case - fix #7297 2023-03-20 09:54:41 -04:00
Gabriel Horner 6108aa1036 Add tests for timestamp blocks
Also make block attributes more explicit in pre-block and reduce needless coupling
2023-02-27 13:20:10 -05:00
Gabriel Horner 6506c369e8 Add support for block refs in page/block props 2023-02-27 13:20:10 -05:00
Tienson Qin 40ad524443
fix: duplicated block ids in multiple files (#8668)
fix: duplicate block ids in multiple files
Co-authored-by: Gabriel Horner <gabriel@logseq.com>
2023-02-21 15:50:09 +08:00
Tienson Qin 3372b3bfbd
fix: duplicate block ids in same file (#8657)
fix: duplicate block ids in the same file
2023-02-21 14:40:43 +08:00
sallto 351a4f34a3
feat: support Media Fragment URIs with audio (#8300)
* fix: fragment urls for audio player

* fix: add Support for url queries and fragments to extension detection

* feat: add tests for extraction of file extension

Co-authored-by: Gabriel Horner <gabriel@logseq.com>
2023-02-06 16:47:23 +08:00
Gabriel Horner 917a8aa098 Fix page blocks not displaying full property values in query tables
We were only saving these in pre-block but we also need them in page
block. Also adds tests for blocks that had this behavior since #6529
2023-02-01 16:28:42 -05:00
Gabriel Horner 78506f22d0 Fix deftests that have confusing args that only apply to defn
Follow up to #8412 where we're accidentally copying
invalid forms from other tests
2023-01-30 10:56:06 -05:00
Brooks Rady e7b511dc5b
fix: normalize the case of property keys (#8412)
* fix: normalize the case of property keys

* chore: remove debugging print statement

* chore: remove unneeded file from diff

* Added tests for Org title parsing

---------

Co-authored-by: Bad3r <bad3r@protonmail.com>
Co-authored-by: Gabriel Horner <97210743+logseq-cldwalker@users.noreply.github.com>
2023-01-30 10:44:23 -05:00
Gabriel Horner e2fe300da7 Move delete-blocks-fn to graph-parser for reuse for nbb
Also:
- Move test to a more appropriate ns - model-test isn't for testing
  higher level parse file behavior
- Delete model fns that are no longer used
- Fix tests which had incorrect target-page-content and were no longer
  testing retractAttribute
- Add options to cli ns for related nbb reuse
- Light cleanup of block deletion
2022-11-26 01:01:36 -05:00
Gabriel Horner 93dd107b8f Fix file links creating garbage pages
Fixes #6034, #6618, #6930
2022-10-27 20:56:31 +08:00
Gabriel Horner 5962e0621d Bump to latest kondo and enable new aliased-namespace-symbol linter 2022-10-19 21:24:54 +08:00
Gabriel Horner daf654172c Disallow tags as property names - fix #6753 2022-10-19 21:19:26 +08:00
Gabriel Horner c6a3eb2de0 Fix title and other built-in properties incorrectly parsing refs
Fixes issues in #6970. Introduced built-in property types as there are
fewer to list as parsed than unparsed.  Also removed test case that was
incorrectly introduced in #5580
2022-10-16 12:17:11 +08:00
Junyi Du 99dc9c371c test: double queare bracket in title 2022-10-16 12:17:11 +08:00
Peng Xiao cca1d35510 Merge remote-tracking branch 'upstream/master' into whiteboards 2022-10-08 17:26:45 +08:00
Junyi Du 0e4037ab61
Feat: new file name escaping rules (#6134)
* feat: new file name escaping for namespace

feat: new file name decoding back to page name

* test: file name sanitization

feat: use _0x to encode %

feat: don't create title property

test: extra URL encoding for escaped file names

fix: fit pdf prefix into new file name rules

fix: encoding rules on some characters

fix: handle the buggy file names imported by users

fix: pdf block ref failed to jump

fix: #6167

* fix: enhance backward compatibility

chore: title validation

test: fix namespace queries test

chore: use index version stored in config.edn instead of search.versions

* feat: convert old version graph mechanism

ui: file conversion UI

feat: rename files for conversion

feat: don't trigger conversion when title property is manually edited

fix: file conflict notification while renaming files on some OS

feat: re-index on update version

feat: clicking NO in the re-index dialog would update the index-ver flag to suppress the dialog

feat: use html entities for reserved char escaping

dev: remove unresolved vars & minor refactor

chore: move file name sanity from gp-util to fs-util, as it's for encoding only but not parsing

test: update file name tests to html entities rule

test: convert files from dir ver 3 for repo_tests

feat: convert Windows reserved file names

fix: save index version into idb instead of file

fix: decode uri of path while parsing files on mobile

fix: couple dir version and index version to ensure only re-index on converted dirs

feat: go back to url-encode for special chars

* chore: fix lint

chore: improve codebase to address Gabriel's comments

fix: remove file remnants on add conflict

fix: remove file remnants on rename conflict

chore: add test ns to nbb runner

Also fix typoed fn and remove unused code

* fix: issues of rebase PR6134 to master after file-sync merged

* feat: switchable filename format

* fix: use  go block to replace promesa for rename all with blocking

* feat: re-index after apply rename all

* ui: file conversion enhancement

* fix: merging filename format PR with master

* fix: filename format lint & CI

* ui: filename format flow

* fix: error handling on the rare internal file path confliction case

* chore: shorten component code for files-breaking-changed

* chore: fix CI

* Minor fixes per latest code review

- Remove unused page-name-order
- Update catch usage to be consistent with what's on master
- Place state fn in right place
- Wording fixes:
  - select and apply -> manual. There are no checkboxes for the user
  - Update -> Edit. We use edit for all other settings button
  - Alternatives to starting sentences with May. Not a common way to
    start a sentence
  - update outdated template comment

* ux: rename instruction update

* ux: rename instruction update (2)

* Tweak wording of conversion modal

Simplifed first paragraph and explained the page to the user in first
sentence, may isn't a common way to start sentences and updated outdated
wording

* Fix large-var warning by splitting out a piece of component

* fix: right slash on Windows; legacy format file sanitization

* fix: triple lowbar renaming fns

Co-authored-by: Gabriel Horner <gabriel@logseq.com>
2022-10-08 15:47:45 +08:00
Tienson Qin 91dddd7541
Fix: stuck re-index when there're multiple whiteboards have the same UUID (#6890)
* fix: keep parsing even if some whiteboards can't be transacted

* enhance: notice parse errors

* enhance: instrument parse-and-load-error

* chore: add tests for whiteboards parsing and loading
2022-10-07 12:04:43 +08:00
Peng Xiao d1a7846a3b fix: tests 2022-09-29 00:21:49 +08:00
Peng Xiao ce4b2a4ad1 fix: use :block/type instead of :block/whiteboard? 2022-09-28 18:27:03 +08:00
Peng Xiao fd94862312 Merge remote-tracking branch 'upstream/master' into whiteboards 2022-09-24 09:42:00 +08:00
Tienson Qin 50db6e42a0 fix: extract property refs support mixed values 2022-09-21 16:46:56 +08:00
Tienson Qin 0ede2d1263 fix: lint 2022-09-21 16:07:05 +08:00
Tienson Qin 1fbb364512 Add :block/properties-text-values 2022-09-21 15:06:48 +08:00
Gabriel Horner edd79954c1 Another test and address review feedback that shouldn't be controversial
- Added test for 71514ab36c
- Also removed dependency that is no longer used
2022-09-15 18:01:29 -04:00
Gabriel Horner ecce39d13b Misc cleanup and test improvements
- Remove incorrect carve entry
- Move test-only code for extract-properties to test
- Remove unused if branches in extract-properties
- Move json parsing of get-references into mldoc to be consistent about
  json encapsulation like other mldoc fns
- reorganize relevant unit tests
- add tests for alias, tags and comma separated config properties
- Update incorrect docstring
2022-09-14 23:26:00 -04:00
Tienson Qin 00e13c1fc5 fix: datoms count 2022-09-14 20:45:04 +08:00
Tienson Qin 79af14d75e fix: graph parser CI 2022-09-09 14:30:39 +08:00
Tienson Qin 58839059ac fix: CI 2022-09-09 14:13:35 +08:00
Tienson Qin cab5dc45a2 fix: tag parsing 2022-09-09 13:19:35 +08:00
Tienson Qin 2c7bae1190 fix: , and some chars shouldn't be supported for tags 2022-09-09 12:39:28 +08:00
Tienson Qin 040fd5510f fix: some tests on graph_parser 2022-09-09 12:39:26 +08:00
Tienson Qin c376c408e0 fix: lint warnings in graph_parser 2022-09-09 12:37:57 +08:00
Tienson Qin 336fa578d5 refactor: replace directives(front matter or org) with properties 2022-09-09 12:37:55 +08:00
Tienson Qin d1d1dc50c2 wip: get references from property values 2022-09-09 12:36:54 +08:00
Peng Xiao 91d63b82b9 test: failed test 2022-09-06 23:42:27 +08:00
Peng Xiao 0ff5324c02 Merge remote-tracking branch 'upstream/master' into whiteboards 2022-08-29 10:45:15 +08:00
Gabriel Horner b47826d69d Fix page block not saving invalid-properties 2022-08-27 10:52:29 +08:00
Gabriel Horner 340fb8cf9a Invalid property fix and tests
- Fix validator considering 'foo bar' or a number or any valid edn, a valid keyword
- Rename property keys to names as users don't know what keys are
- Also bumped nbb-logseq to latest version
- Also give property validation more product focused namespace and name
2022-08-27 10:52:29 +08:00
Peng Xiao 37d0ad2524 Merge remote-tracking branch 'upstream/master' into whiteboards 2022-08-24 16:50:57 +08:00
Gabriel Horner 55ccc2d17c Rename config to :rich-property-values?
Name should allow for additional functionality later
2022-08-23 09:55:15 -04:00
Peng Xiao 03ccde7008 fix: ci tests 2022-08-16 17:59:39 +08:00
Peng Xiao 384fc95231 fix: whiteboard name case issue 2022-08-16 17:52:43 +08:00
Gabriel Horner 509422901a Fix bugs with tests
- Quoted property values were extracting refs
- Punctuation at end of tags was getting included in tag
- Property values were only getting displayed as refs/relationships
  instead of full property value
- cli wasn't passing along user config
2022-08-11 17:20:47 -04:00
Gabriel Horner 39b43b363c Fix page-properties bug and dsl-query bug
parser test caught page properties bug and dsl-query tests once written
correctly exposed another bug. Also remove unit test as they are
replaced by integration style tests in graph-parser-test
2022-08-11 16:32:00 -04:00
Gabriel Horner 0719163d30 Split out parse-property for use in dsl-query and tests
- Add tests for all *property dsl queries with and without new
  config option
- Add tests for property persistence
- Add tests for property relationships
2022-08-11 16:32:00 -04:00
Gabriel Horner 405183db09 Add config option to allow for more free text in property values
Still need to test that *property operators work correctly when this
option is on
2022-08-11 16:32:00 -04:00
Peng Xiao b9bc4130e9 fix: tests 2022-08-06 13:29:36 +08:00
Peng Xiao 1e476bd26c feat: add tabler icon extensions 2022-08-05 21:50:58 +08:00
Peng Xiao b30adea656 Merge remote-tracking branch 'upstream/master' into whiteboards 2022-08-05 13:59:06 +08:00
Tienson Qin 59b924f25d fix: CI 2022-08-04 17:25:29 +08:00
Gabriel Horner e0b1f6b2de Move common uses of page-ref to its own ns
By having a specific ns for page-ref utils, our code is more readable
and intention revealing. Also found that
text/{get-page-name,page-ref-un-brackets!} was getting called in
contexts that didn't make sense e.g. query layer was checking for
markdown and org page-refs
2022-08-04 17:25:29 +08:00
Peng Xiao ab6eec5fc3 Merge remote-tracking branch 'upstream/master' into whiteboards 2022-08-03 14:12:10 +08:00
Gabriel Horner 49fbdd0244
Merge pull request #6214 from logseq/fix/hidden-properties-in-autocompletion
Fix: Property completion for hidden properties and property ordering
2022-08-01 14:42:06 -04:00
Peng Xiao 45272ef52b Merge remote-tracking branch 'upstream/master' into whiteboards 2022-08-01 10:22:15 +08:00
Gabriel Horner 42b7f5d189 Add two editable org built-in properties
Also ported old filetags property tests from b991599244.
2022-07-30 06:33:37 +08:00